Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
2026-09-24T19:57:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:57:06Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260924 19:57:06.798856 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.190:46309
--webserver_interface=127.17.2.190
--webserver_port=0
--builtin_ntp_servers=127.17.2.148:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260924 19:57:07.107242 17427 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:57:07.107817 17427 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:57:07.108206 17427 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260924 19:57:07.141647 17427 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:57:07.141945 17427 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:57:07.142160 17427 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:57:07.142361 17427 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:57:07.180850 17427 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.190:46309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.2.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:57:07.182256 17427 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:57:07.183934 17427 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260924 19:57:07.196965 17433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:08.600682 17432 debug-util.cc:398] Leaking SignalData structure 0x7b08000379a0 after lost signal to thread 17427
W20260924 19:57:08.987787 17427 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.791s user 0.655s sys 1.131s
W20260924 19:57:08.988212 17427 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.792s user 0.655s sys 1.131s
W20260924 19:57:07.197338 17434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:08.990834 17436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:08.993474 17435 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1793 milliseconds
I20260924 19:57:08.993482 17427 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260924 19:57:08.994838 17427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:57:08.997308 17427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:57:08.998653 17427 hybrid_clock.cc:648] HybridClock initialized: now 1790279828998617 us; error 65 us; skew 500 ppm
I20260924 19:57:08.999594 17427 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:57:09.006174 17427 webserver.cc:533] Webserver started at http://127.17.2.190:37347/ using document root <none> and password file <none>
I20260924 19:57:09.007314 17427 fs_manager.cc:362] Metadata directory not provided
I20260924 19:57:09.007561 17427 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:57:09.008062 17427 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:57:09.012602 17427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data/instance:
uuid: "3e059bbe4dac4178bccb3be48cb1118d"
format_stamp: "Formatted at 2026-09-24 19:57:09 on dist-test-slave-h376"
I20260924 19:57:09.013777 17427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal/instance:
uuid: "3e059bbe4dac4178bccb3be48cb1118d"
format_stamp: "Formatted at 2026-09-24 19:57:09 on dist-test-slave-h376"
I20260924 19:57:09.021687 17427 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260924 19:57:09.027793 17443 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:09.029130 17427 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260924 19:57:09.029495 17427 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal
uuid: "3e059bbe4dac4178bccb3be48cb1118d"
format_stamp: "Formatted at 2026-09-24 19:57:09 on dist-test-slave-h376"
I20260924 19:57:09.029848 17427 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:57:09.080650 17427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:57:09.082212 17427 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:57:09.082710 17427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:57:09.158221 17427 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.190:46309
I20260924 19:57:09.158322 17494 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.190:46309 every 8 connection(s)
I20260924 19:57:09.161217 17427 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data/info.pb
I20260924 19:57:09.164700 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 17427
I20260924 19:57:09.165282 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal/instance
I20260924 19:57:09.168126 17495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:09.169041 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.189:43689
--webserver_interface=127.17.2.189
--webserver_port=0
--builtin_ntp_servers=127.17.2.148:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260924 19:57:09.179869 17495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:57:09.214845 17495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } has no permanent_uuid. Determining permanent_uuid...
W20260924 19:57:09.217615 17444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.189:43689: connect: Connection refused (error 111)
W20260924 19:57:09.222617 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:43689: Network error: Client connection negotiation failed: client connection to 127.17.2.189:43689: connect: Connection refused (error 111)
I20260924 19:57:09.272285 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } attempt: 1
W20260924 19:57:09.277801 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:43689: Network error: Client connection negotiation failed: client connection to 127.17.2.189:43689: connect: Connection refused (error 111)
I20260924 19:57:09.346230 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } attempt: 2
W20260924 19:57:09.351464 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:43689: Network error: Client connection negotiation failed: client connection to 127.17.2.189:43689: connect: Connection refused (error 111)
I20260924 19:57:09.442909 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } attempt: 3
W20260924 19:57:09.447876 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:43689: Network error: Client connection negotiation failed: client connection to 127.17.2.189:43689: connect: Connection refused (error 111)
W20260924 19:57:09.512393 17497 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:57:09.513072 17497 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:57:09.513561 17497 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260924 19:57:09.547071 17497 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:57:09.547528 17497 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:57:09.547819 17497 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:57:09.548092 17497 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:57:09.587769 17497 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.189:43689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.2.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:57:09.589267 17497 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:57:09.591287 17497 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260924 19:57:09.591343 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } attempt: 4
W20260924 19:57:09.595683 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:43689: Network error: Client connection negotiation failed: client connection to 127.17.2.189:43689: connect: Connection refused (error 111)
W20260924 19:57:09.604352 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:57:09.895078 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } attempt: 5
W20260924 19:57:09.903471 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:43689: Network error: Client connection negotiation failed: client connection to 127.17.2.189:43689: connect: Connection refused (error 111)
I20260924 19:57:10.451108 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } attempt: 6
W20260924 19:57:10.483916 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:43689: Network error: Client connection negotiation failed: client connection to 127.17.2.189:43689: connect: Connection refused (error 111)
W20260924 19:57:11.007856 17505 debug-util.cc:398] Leaking SignalData structure 0x7b08000379a0 after lost signal to thread 17497
W20260924 19:57:09.604951 17507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:11.117456 17497 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.513s user 0.455s sys 1.057s
W20260924 19:57:11.118353 17497 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.514s user 0.455s sys 1.058s
W20260924 19:57:11.120124 17509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:11.122064 17508 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1513 milliseconds
I20260924 19:57:11.122143 17497 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260924 19:57:11.123555 17497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:57:11.125957 17497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:57:11.127274 17497 hybrid_clock.cc:648] HybridClock initialized: now 1790279831127238 us; error 71 us; skew 500 ppm
I20260924 19:57:11.128129 17497 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:57:11.134382 17497 webserver.cc:533] Webserver started at http://127.17.2.189:38727/ using document root <none> and password file <none>
I20260924 19:57:11.135416 17497 fs_manager.cc:362] Metadata directory not provided
I20260924 19:57:11.135658 17497 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:57:11.136129 17497 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:57:11.140582 17497 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/data/instance:
uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6"
format_stamp: "Formatted at 2026-09-24 19:57:11 on dist-test-slave-h376"
I20260924 19:57:11.141768 17497 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/wal/instance:
uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6"
format_stamp: "Formatted at 2026-09-24 19:57:11 on dist-test-slave-h376"
I20260924 19:57:11.149616 17497 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260924 19:57:11.156013 17518 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:11.157420 17497 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260924 19:57:11.157860 17497 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/wal
uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6"
format_stamp: "Formatted at 2026-09-24 19:57:11 on dist-test-slave-h376"
I20260924 19:57:11.158311 17497 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:57:11.217213 17497 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:57:11.218660 17497 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:57:11.219069 17497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:57:11.291453 17497 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.189:43689
I20260924 19:57:11.291544 17569 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.189:43689 every 8 connection(s)
I20260924 19:57:11.294272 17497 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/data/info.pb
I20260924 19:57:11.297500 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 17497
I20260924 19:57:11.298022 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-1/wal/instance
I20260924 19:57:11.301251 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.188:38761
--webserver_interface=127.17.2.188
--webserver_port=0
--builtin_ntp_servers=127.17.2.148:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260924 19:57:11.301728 17570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:11.313900 17570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:57:11.336373 17570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:57:11.351276 17570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } has no permanent_uuid. Determining permanent_uuid...
W20260924 19:57:11.353749 17521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
W20260924 19:57:11.358238 17570 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
I20260924 19:57:11.407749 17570 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 1
W20260924 19:57:11.412055 17570 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
I20260924 19:57:11.480485 17570 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 2
W20260924 19:57:11.485049 17570 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
I20260924 19:57:11.544528 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } attempt: 7
I20260924 19:57:11.556460 17495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } has no permanent_uuid. Determining permanent_uuid...
W20260924 19:57:11.562247 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
I20260924 19:57:11.576458 17570 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 3
W20260924 19:57:11.580623 17570 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
I20260924 19:57:11.620729 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 1
W20260924 19:57:11.625038 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
W20260924 19:57:11.630278 17572 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:57:11.630904 17572 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:57:11.631412 17572 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260924 19:57:11.664691 17572 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:57:11.665092 17572 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:57:11.665411 17572 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:57:11.665644 17572 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:57:11.702965 17572 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.188:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.2.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:57:11.704290 17572 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:57:11.705886 17572 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260924 19:57:11.706466 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 2
W20260924 19:57:11.710851 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
W20260924 19:57:11.718389 17584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:57:11.724030 17570 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 4
W20260924 19:57:11.730146 17570 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
I20260924 19:57:11.796258 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 3
W20260924 19:57:11.800808 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
I20260924 19:57:11.941231 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 4
W20260924 19:57:11.948458 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
I20260924 19:57:12.029788 17570 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 5
W20260924 19:57:12.037070 17570 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
I20260924 19:57:12.232031 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 5
W20260924 19:57:12.239861 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
I20260924 19:57:12.584702 17570 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 6
W20260924 19:57:12.596284 17570 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
I20260924 19:57:12.792443 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 6
W20260924 19:57:12.802735 17495 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:38761: Network error: Client connection negotiation failed: client connection to 127.17.2.188:38761: connect: Connection refused (error 111)
W20260924 19:57:11.719662 17585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:13.010535 17587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:13.014393 17586 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1288 milliseconds
I20260924 19:57:13.014487 17572 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260924 19:57:13.016144 17572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:57:13.019614 17572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:57:13.021063 17572 hybrid_clock.cc:648] HybridClock initialized: now 1790279833020973 us; error 77 us; skew 500 ppm
I20260924 19:57:13.021996 17572 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:57:13.029824 17572 webserver.cc:533] Webserver started at http://127.17.2.188:39525/ using document root <none> and password file <none>
I20260924 19:57:13.031008 17572 fs_manager.cc:362] Metadata directory not provided
I20260924 19:57:13.031373 17572 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:57:13.031863 17572 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:57:13.036696 17572 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/data/instance:
uuid: "e1ff67c5d57d474a9d92cc4c6de1287e"
format_stamp: "Formatted at 2026-09-24 19:57:13 on dist-test-slave-h376"
I20260924 19:57:13.037773 17572 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/wal/instance:
uuid: "e1ff67c5d57d474a9d92cc4c6de1287e"
format_stamp: "Formatted at 2026-09-24 19:57:13 on dist-test-slave-h376"
I20260924 19:57:13.046272 17572 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20260924 19:57:13.053102 17598 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:13.054517 17572 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:57:13.054885 17572 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/wal
uuid: "e1ff67c5d57d474a9d92cc4c6de1287e"
format_stamp: "Formatted at 2026-09-24 19:57:13 on dist-test-slave-h376"
I20260924 19:57:13.055271 17572 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:57:13.108613 17572 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:57:13.110097 17572 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:57:13.110582 17572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:57:13.189517 17572 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.188:38761
I20260924 19:57:13.189595 17649 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.188:38761 every 8 connection(s)
I20260924 19:57:13.192615 17572 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/data/info.pb
I20260924 19:57:13.194906 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 17572
I20260924 19:57:13.196447 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-2/wal/instance
I20260924 19:57:13.200492 17650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:13.216234 17650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:57:13.239306 17650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:57:13.252703 17650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:57:13.271641 17650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Bootstrap starting.
I20260924 19:57:13.278844 17650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:13.280880 17650 log.cc:826] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:57:13.285668 17650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: No bootstrap required, opened a new log
I20260924 19:57:13.314489 17650 raft_consensus.cc:359] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:13.315478 17650 raft_consensus.cc:385] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:13.315841 17650 raft_consensus.cc:740] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ff67c5d57d474a9d92cc4c6de1287e, State: Initialized, Role: FOLLOWER
I20260924 19:57:13.316920 17650 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:13.319964 17660 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:13.320643 17660 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:13.321913 17650 sys_catalog.cc:565] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:57:13.349748 17660 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:13.350348 17660 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
W20260924 19:57:13.351495 17671 catalog_manager.cc:1607] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:57:13.351794 17671 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:57:13.353322 17660 leader_election.cc:290] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689)
W20260924 19:57:13.365916 17604 tablet.cc:2433] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:57:13.345762 17572 master_runner.cc:187] Error getting master registration for 127.17.2.190:46309: OK, instance_id { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" instance_seqno: 1790279829136267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:57:13.377986 17572 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:57:13.656802 17570 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 7
I20260924 19:57:13.679805 17570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6: Bootstrap starting.
I20260924 19:57:13.689728 17570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:13.694156 17570 log.cc:826] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:57:13.705348 17570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6: No bootstrap required, opened a new log
I20260924 19:57:13.733280 17570 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:13.734264 17570 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:13.734586 17570 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f88e00b235f48bb82e6a3c9c9f51fa6, State: Initialized, Role: FOLLOWER
I20260924 19:57:13.735579 17570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:13.739459 17675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:13.740348 17675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:13.741446 17570 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:57:13.741667 17545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:13.742535 17545 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ff67c5d57d474a9d92cc4c6de1287e in term 0.
I20260924 19:57:13.744154 17599 leader_election.cc:304] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e; no voters:
I20260924 19:57:13.745042 17660 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:13.745411 17660 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:13.745725 17660 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:13.781877 17686 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:57:13.782356 17686 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:57:13.793309 17660 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:13.795506 17660 leader_election.cc:290] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 1 election: Requested vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689)
I20260924 19:57:13.801081 17545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6"
I20260924 19:57:13.801740 17545 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:13.777699 17497 master_runner.cc:187] Error getting master registration for 127.17.2.190:46309: OK, instance_id { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" instance_seqno: 1790279829136267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:57:13.809450 17497 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:57:13.836192 17495 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } attempt: 7
I20260924 19:57:13.861034 17495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Bootstrap starting.
I20260924 19:57:13.868934 17495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:13.871347 17495 log.cc:826] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:57:13.876672 17495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: No bootstrap required, opened a new log
I20260924 19:57:13.905546 17545 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ff67c5d57d474a9d92cc4c6de1287e in term 1.
I20260924 19:57:13.907949 17599 leader_election.cc:304] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e; no voters:
I20260924 19:57:13.908841 17660 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:13.912914 17495 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:13.913962 17495 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:13.914350 17495 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e059bbe4dac4178bccb3be48cb1118d, State: Initialized, Role: FOLLOWER
I20260924 19:57:13.915652 17495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:13.931819 17660 raft_consensus.cc:697] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 1 LEADER]: Becoming Leader. State: Replica: e1ff67c5d57d474a9d92cc4c6de1287e, State: Running, Role: LEADER
I20260924 19:57:13.932349 17691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:13.933310 17691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:13.934057 17660 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:13.934901 17495 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:57:13.935329 17469 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d"
I20260924 19:57:13.936800 17469 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:13.936048 17470 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
W20260924 19:57:13.944463 17470 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35334 (request call id 2) took 590 ms (client timeout 619 ms). Trace:
W20260924 19:57:13.944890 17470 rpcz_store.cc:269] 0924 19:57:13.353845 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:13.354004 (+ 159us) service_pool.cc:224] Handling call
0924 19:57:13.944380 (+590376us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:13.957476 17690 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ff67c5d57d474a9d92cc4c6de1287e. Latest consensus state: current_term: 1 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:13.958278 17690 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:13.961457 17697 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260924 19:57:13.970465 17697 catalog_manager.cc:1533] Initializing Kudu cluster ID...
W20260924 19:57:13.977334 17524 tablet.cc:2433] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:57:13.994053 17545 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Refusing update from remote peer e1ff67c5d57d474a9d92cc4c6de1287e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:57:13.997890 17469 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ff67c5d57d474a9d92cc4c6de1287e in term 1.
I20260924 19:57:13.999528 17690 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:14.018834 17469 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Refusing update from remote peer e1ff67c5d57d474a9d92cc4c6de1287e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:57:14.021102 17690 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:14.026503 17675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ff67c5d57d474a9d92cc4c6de1287e. Latest consensus state: current_term: 1 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:14.027824 17675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:14.032207 17699 mvcc.cc:204] Tried to move back new op lower bound from 7332986200021102592 to 7332986199814877184. Current Snapshot: MvccSnapshot[applied={T|T < 7332986200021102592}]
I20260924 19:57:14.060362 17675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:14.061379 17675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:14.074594 17697 catalog_manager.cc:1396] Generated new cluster ID: 08019495472c40618d8cccdc88d885cc
I20260924 19:57:14.074934 17697 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260924 19:57:14.078095 17690 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:14.078788 17690 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: LEADER
W20260924 19:57:14.094166 17449 tablet.cc:2433] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:57:14.094960 17660 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:14.095723 17660 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:14.095355 17691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ff67c5d57d474a9d92cc4c6de1287e. Latest consensus state: current_term: 1 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:14.096275 17691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:14.100270 17704 mvcc.cc:204] Tried to move back new op lower bound from 7332986200021102592 to 7332986199814877184. Current Snapshot: MvccSnapshot[applied={T|T < 7332986200021102592}]
I20260924 19:57:14.126318 17691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:14.128302 17691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:14.149034 17697 catalog_manager.cc:1419] Generated new certificate authority record
I20260924 19:57:14.151093 17697 catalog_manager.cc:1553] Loading token signing keys...
I20260924 19:57:14.209900 17697 catalog_manager.cc:6219] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Generated new TSK 0
I20260924 19:57:14.211146 17697 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260924 19:57:14.215621 17712 catalog_manager.cc:1308] Loaded cluster ID: 08019495472c40618d8cccdc88d885cc
I20260924 19:57:14.215981 17712 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: loading cluster ID for follower catalog manager: success
I20260924 19:57:14.233402 17712 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: acquiring CA information for follower catalog manager: success
I20260924 19:57:14.262187 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.2.129:0
--local_ip_for_outbound_sockets=127.17.2.129
--webserver_interface=127.17.2.129
--webserver_port=0
--tserver_master_addrs=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--builtin_ntp_servers=127.17.2.148:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260924 19:57:14.274876 17712 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:57:14.788250 17686 catalog_manager.cc:1308] Loaded cluster ID: 08019495472c40618d8cccdc88d885cc
I20260924 19:57:14.788538 17686 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6: loading cluster ID for follower catalog manager: success
I20260924 19:57:14.796630 17686 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6: acquiring CA information for follower catalog manager: success
I20260924 19:57:14.809932 17686 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 19:57:14.886902 17714 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:57:14.887642 17714 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:57:14.888525 17714 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260924 19:57:14.952764 17714 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:57:14.954366 17714 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.2.129
I20260924 19:57:15.025854 17714 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.2.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.2.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.2.129
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:57:15.027691 17714 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:57:15.030256 17714 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260924 19:57:15.060927 17720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:57:15.126580 17725 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:15.131430 17725 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:15.135496 17470 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:15.143699 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:15.145738 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:15.151199 17725 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:15.151970 17727 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:15.184737 17728 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:15.185338 17728 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:15.196952 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:15.197081 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:15.199503 17444 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:15.203837 17728 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:15.204421 17728 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:16.213979 17730 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:16.223529 17730 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:16.226898 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:16.232348 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:16.234627 17446 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:16.244279 17730 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:16.245393 17731 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:16.487848 17732 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:16.495396 17732 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:16.498656 17732 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:16.499303 17470 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:16.503924 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:16.506439 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:16.508219 17732 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:16.793505 17734 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:16.797835 17734 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:16.801425 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:16.809808 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:16.811686 17446 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:16.816939 17734 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:16.818377 17734 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:17.114794 17736 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:17.115698 17736 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:17.127947 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:17.129755 17736 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:17.152755 17470 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:17.155239 17522 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:17.157362 17736 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:17.718305 17738 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:17.718916 17738 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:17.721813 17470 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:17.724915 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:17.726572 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:17.729317 17738 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:17.730316 17738 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:17.757704 17740 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:17.758530 17740 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:17.761677 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:17.762262 17544 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e059bbe4dac4178bccb3be48cb1118d in term 1.
I20260924 19:57:17.763864 17444 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d, 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters:
I20260924 19:57:17.770617 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:17.772965 17740 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:17.773921 17740 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:57:17.774732 17740 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:17.775450 17740 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:57:17.783474 17740 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:17.786289 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6"
I20260924 19:57:17.786821 17544 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:57:17.793852 17544 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e059bbe4dac4178bccb3be48cb1118d in term 2.
I20260924 19:57:17.795195 17444 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d, 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters:
I20260924 19:57:17.796125 17741 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:57:17.803910 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e"
I20260924 19:57:17.805717 17741 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 2 LEADER]: Becoming Leader. State: Replica: 3e059bbe4dac4178bccb3be48cb1118d, State: Running, Role: LEADER
I20260924 19:57:17.806805 17741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:17.809688 17740 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 2 election: Requested vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:17.821470 17544 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 2 FOLLOWER]: Rejecting Update request from peer e1ff67c5d57d474a9d92cc4c6de1287e for earlier term 1. Current term is 2. Ops: []
I20260924 19:57:17.823437 17743 consensus_queue.cc:1059] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260924 19:57:17.829789 17742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e059bbe4dac4178bccb3be48cb1118d. Latest consensus state: current_term: 2 leader_uuid: "3e059bbe4dac4178bccb3be48cb1118d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:17.830674 17742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:17.837700 17746 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260924 19:57:17.842839 17745 raft_consensus.cc:3055] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 1 LEADER]: Stepping down as leader of term 1
I20260924 19:57:17.843246 17745 raft_consensus.cc:740] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e1ff67c5d57d474a9d92cc4c6de1287e, State: Running, Role: LEADER
I20260924 19:57:17.844139 17745 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:17.865316 17745 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:57:17.877525 17746 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260924 19:57:17.888445 17746 catalog_manager.cc:1308] Loaded cluster ID: 08019495472c40618d8cccdc88d885cc
I20260924 19:57:17.902189 17746 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260924 19:57:17.909658 17746 catalog_manager.cc:1553] Loading token signing keys...
I20260924 19:57:17.925370 17746 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Loaded TSK: 0
I20260924 19:57:17.927425 17746 catalog_manager.cc:1563] Initializing in-progress tserver states...
W20260924 19:57:15.075700 17721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:18.114604 17722 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 3043 milliseconds
W20260924 19:57:16.453670 17719 debug-util.cc:398] Leaking SignalData structure 0x7b0800037240 after lost signal to thread 17714
W20260924 19:57:18.113842 17714 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 3.043s user 0.607s sys 1.121s
W20260924 19:57:18.115775 17714 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 3.046s user 0.607s sys 1.121s
I20260924 19:57:18.116834 17714 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20260924 19:57:18.116928 17724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:57:18.120421 17714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:57:18.122840 17714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:57:18.124265 17714 hybrid_clock.cc:648] HybridClock initialized: now 1790279838124222 us; error 49 us; skew 500 ppm
I20260924 19:57:18.125156 17714 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:57:18.132259 17714 webserver.cc:533] Webserver started at http://127.17.2.129:44635/ using document root <none> and password file <none>
I20260924 19:57:18.133502 17714 fs_manager.cc:362] Metadata directory not provided
I20260924 19:57:18.133798 17714 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:57:18.134312 17714 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:57:18.139379 17714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/data/instance:
uuid: "c4f6e3a7b06148918d230c77ab0c8202"
format_stamp: "Formatted at 2026-09-24 19:57:18 on dist-test-slave-h376"
I20260924 19:57:18.140618 17714 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/instance:
uuid: "c4f6e3a7b06148918d230c77ab0c8202"
format_stamp: "Formatted at 2026-09-24 19:57:18 on dist-test-slave-h376"
I20260924 19:57:18.149084 17714 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260924 19:57:18.155877 17752 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:18.157241 17714 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260924 19:57:18.157639 17714 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal
uuid: "c4f6e3a7b06148918d230c77ab0c8202"
format_stamp: "Formatted at 2026-09-24 19:57:18 on dist-test-slave-h376"
I20260924 19:57:18.157997 17714 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:57:18.245512 17714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:57:18.247649 17714 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:57:18.248222 17714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:57:18.251886 17714 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260924 19:57:18.252772 17745 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:18.253216 17745 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:18.254908 17745 leader_election.cc:290] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689)
I20260924 19:57:18.255796 17470 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:18.256460 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:18.257050 17544 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ff67c5d57d474a9d92cc4c6de1287e in term 2.
I20260924 19:57:18.258320 17599 leader_election.cc:304] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e; no voters: 3e059bbe4dac4178bccb3be48cb1118d
I20260924 19:57:18.258865 17714 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:57:18.259143 17714 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:18.259047 17745 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 19:57:18.259492 17714 ts_tablet_manager.cc:616] Registered 0 tablets
I20260924 19:57:18.259474 17745 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:18.259704 17714 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:18.259760 17745 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:57:18.265525 17745 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:18.267997 17470 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d"
I20260924 19:57:18.268155 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6"
I20260924 19:57:18.268666 17544 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:57:18.271304 17745 leader_election.cc:290] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 3 election: Requested vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689)
I20260924 19:57:18.274681 17544 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ff67c5d57d474a9d92cc4c6de1287e in term 3.
I20260924 19:57:18.275784 17599 leader_election.cc:304] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e; no voters: 3e059bbe4dac4178bccb3be48cb1118d
I20260924 19:57:18.277117 17745 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:57:18.277643 17745 raft_consensus.cc:697] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 3 LEADER]: Becoming Leader. State: Replica: e1ff67c5d57d474a9d92cc4c6de1287e, State: Running, Role: LEADER
I20260924 19:57:18.278532 17745 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:18.284704 17747 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ff67c5d57d474a9d92cc4c6de1287e. Latest consensus state: current_term: 3 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:18.285429 17747 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:18.287498 17768 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260924 19:57:18.294900 17768 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260924 19:57:18.299412 17768 catalog_manager.cc:1308] Loaded cluster ID: 08019495472c40618d8cccdc88d885cc
I20260924 19:57:18.299731 17768 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260924 19:57:18.304694 17768 catalog_manager.cc:1553] Loading token signing keys...
I20260924 19:57:18.308974 17768 catalog_manager.cc:6230] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Loaded TSK: 0
I20260924 19:57:18.310274 17768 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260924 19:57:18.393764 17572 master_runner.cc:233] Existing masters have differing terms: 2 vs 3
I20260924 19:57:18.394299 17572 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:57:18.400856 17544 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Rejecting Update request from peer 3e059bbe4dac4178bccb3be48cb1118d for earlier term 2. Current term is 3. Ops: []
I20260924 19:57:18.403261 17820 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 19:57:18.405407 17827 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 2 LEADER]: Stepping down as leader of term 2
I20260924 19:57:18.405757 17827 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3e059bbe4dac4178bccb3be48cb1118d, State: Running, Role: LEADER
I20260924 19:57:18.406582 17827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:18.407784 17827 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:57:18.408522 17625 raft_consensus.cc:1240] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 3 LEADER]: Rejecting Update request from peer 3e059bbe4dac4178bccb3be48cb1118d for earlier term 2. Current term is 3. Ops: []
I20260924 19:57:18.479050 17714 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.129:42465
I20260924 19:57:18.479240 17869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.129:42465 every 8 connection(s)
I20260924 19:57:18.481765 17714 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/data/info.pb
I20260924 19:57:18.486714 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 17714
I20260924 19:57:18.487458 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/instance
I20260924 19:57:18.510565 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.2.130:0
--local_ip_for_outbound_sockets=127.17.2.130
--webserver_interface=127.17.2.130
--webserver_port=0
--tserver_master_addrs=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--builtin_ntp_servers=127.17.2.148:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260924 19:57:18.573768 17870 heartbeater.cc:344] Connected to a master server at 127.17.2.188:38761
I20260924 19:57:18.574337 17870 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:18.575641 17870 heartbeater.cc:507] Master 127.17.2.188:38761 requested a full tablet report, sending...
I20260924 19:57:18.578675 17615 ts_manager.cc:194] Registered new tserver with Master: c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:18.581377 17615 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.17.2.129:36093
I20260924 19:57:18.589337 17871 heartbeater.cc:344] Connected to a master server at 127.17.2.189:43689
I20260924 19:57:18.589694 17871 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:18.590369 17871 heartbeater.cc:507] Master 127.17.2.189:43689 requested a full tablet report, sending...
I20260924 19:57:18.593628 17535 ts_manager.cc:194] Registered new tserver with Master: c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:18.616035 17872 heartbeater.cc:344] Connected to a master server at 127.17.2.190:46309
I20260924 19:57:18.616387 17872 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:18.616961 17872 heartbeater.cc:507] Master 127.17.2.190:46309 requested a full tablet report, sending...
I20260924 19:57:18.619697 17460 ts_manager.cc:194] Registered new tserver with Master: c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:18.725149 17544 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Refusing update from remote peer e1ff67c5d57d474a9d92cc4c6de1287e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260924 19:57:18.726486 17747 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 19:57:18.732384 17881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ff67c5d57d474a9d92cc4c6de1287e. Latest consensus state: current_term: 3 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:18.733183 17881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:18.738513 17470 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Refusing update from remote peer e1ff67c5d57d474a9d92cc4c6de1287e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260924 19:57:18.738205 17747 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:18.738914 17470 pending_rounds.cc:85] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Aborting all ops after (but not including) 4
I20260924 19:57:18.739059 17747 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:18.739214 17470 pending_rounds.cc:107] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260924 19:57:18.739501 17470 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 19:57:18.738816 17881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:18.740396 17881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:18.741039 17745 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260924 19:57:18.748821 17825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:18.749589 17825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:18.749631 17827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ff67c5d57d474a9d92cc4c6de1287e. Latest consensus state: current_term: 3 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:18.750505 17827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:57:18.887212 17877 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:57:18.887706 17877 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:57:18.888228 17877 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260924 19:57:18.922278 17877 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:57:18.923177 17877 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.2.130
I20260924 19:57:18.965341 17877 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.2.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.2.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.2.130
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:57:18.966997 17877 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:57:18.968730 17877 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260924 19:57:18.981942 17891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:57:19.085723 17870 heartbeater.cc:499] Master 127.17.2.188:38761 was elected leader, sending a full tablet report...
I20260924 19:57:19.153236 17745 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:19.172415 17745 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:19.236469 17881 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:19.237020 17881 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:19.239009 17881 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:19.241576 17470 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:19.241644 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:19.243685 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:19.244798 17881 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 19:57:20.161985 17898 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:20.163323 17898 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:20.168589 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:20.174027 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:20.178648 17446 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:20.181270 17898 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:20.181438 17899 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 19:57:20.384727 17890 debug-util.cc:398] Leaking SignalData structure 0x7b0800037240 after lost signal to thread 17877
W20260924 19:57:20.622036 17890 kernel_stack_watchdog.cc:198] Thread 17877 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260924 19:57:18.982383 17892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:20.623299 17877 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.641s user 0.543s sys 0.968s
W20260924 19:57:20.623657 17877 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.642s user 0.543s sys 0.968s
W20260924 19:57:20.626092 17894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:20.629503 17893 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1646 milliseconds
I20260924 19:57:20.629508 17877 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260924 19:57:20.630848 17877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:57:20.633035 17877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:57:20.634392 17877 hybrid_clock.cc:648] HybridClock initialized: now 1790279840634360 us; error 49 us; skew 500 ppm
I20260924 19:57:20.635460 17877 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:57:20.642217 17877 webserver.cc:533] Webserver started at http://127.17.2.130:45483/ using document root <none> and password file <none>
I20260924 19:57:20.643494 17877 fs_manager.cc:362] Metadata directory not provided
I20260924 19:57:20.643700 17877 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:57:20.644116 17877 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:57:20.650211 17877 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/data/instance:
uuid: "f74e6ed351284d7fb110e44336ce643a"
format_stamp: "Formatted at 2026-09-24 19:57:20 on dist-test-slave-h376"
I20260924 19:57:20.651454 17877 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/instance:
uuid: "f74e6ed351284d7fb110e44336ce643a"
format_stamp: "Formatted at 2026-09-24 19:57:20 on dist-test-slave-h376"
I20260924 19:57:20.659832 17877 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.001s
I20260924 19:57:20.666834 17904 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:20.668098 17877 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 19:57:20.668485 17877 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal
uuid: "f74e6ed351284d7fb110e44336ce643a"
format_stamp: "Formatted at 2026-09-24 19:57:20 on dist-test-slave-h376"
I20260924 19:57:20.668864 17877 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:57:20.720511 17877 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:57:20.722172 17877 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:57:20.722667 17877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:57:20.725735 17877 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260924 19:57:20.730186 17877 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:57:20.730525 17877 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:20.730868 17877 ts_tablet_manager.cc:616] Registered 0 tablets
I20260924 19:57:20.731142 17877 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:20.882288 17877 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.130:41301
I20260924 19:57:20.882403 18016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.130:41301 every 8 connection(s)
I20260924 19:57:20.885232 17877 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/data/info.pb
I20260924 19:57:20.894802 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 17877
I20260924 19:57:20.895560 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/instance
I20260924 19:57:20.921448 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.2.131:0
--local_ip_for_outbound_sockets=127.17.2.131
--webserver_interface=127.17.2.131
--webserver_port=0
--tserver_master_addrs=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--builtin_ntp_servers=127.17.2.148:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260924 19:57:20.941207 18018 heartbeater.cc:344] Connected to a master server at 127.17.2.189:43689
I20260924 19:57:20.941735 18018 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:20.943181 18019 heartbeater.cc:344] Connected to a master server at 127.17.2.190:46309
I20260924 19:57:20.943122 18018 heartbeater.cc:507] Master 127.17.2.189:43689 requested a full tablet report, sending...
I20260924 19:57:20.943527 18019 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:20.944461 18019 heartbeater.cc:507] Master 127.17.2.190:46309 requested a full tablet report, sending...
I20260924 19:57:20.946617 17535 ts_manager.cc:194] Registered new tserver with Master: f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:20.947351 17460 ts_manager.cc:194] Registered new tserver with Master: f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:20.950500 18017 heartbeater.cc:344] Connected to a master server at 127.17.2.188:38761
I20260924 19:57:20.950852 18017 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:20.951792 18017 heartbeater.cc:507] Master 127.17.2.188:38761 requested a full tablet report, sending...
I20260924 19:57:20.953825 17615 ts_manager.cc:194] Registered new tserver with Master: f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:20.955129 17615 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.17.2.130:37869
I20260924 19:57:21.167927 18028 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:21.168443 18028 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:21.171376 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:21.171671 18028 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:21.172441 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:21.173986 17446 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:21.174782 18028 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 19:57:21.269007 18027 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:57:21.269552 18027 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:57:21.270139 18027 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260924 19:57:21.304438 18027 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:57:21.305415 18027 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.2.131
I20260924 19:57:21.343860 18027 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.2.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.2.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.2.131
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:57:21.345320 18027 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:57:21.347035 18027 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260924 19:57:21.360890 18034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:57:21.458837 18017 heartbeater.cc:499] Master 127.17.2.188:38761 was elected leader, sending a full tablet report...
W20260924 19:57:21.477787 17566 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1260 after lost signal to thread 17501
I20260924 19:57:21.773444 18041 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:21.774504 18041 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:21.778780 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:21.780923 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:21.787650 17446 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:21.793313 18041 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:21.794724 18041 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 19:57:22.378665 18043 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:22.380698 18043 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:22.384933 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:22.387974 18043 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:22.389190 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:22.390964 17446 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:22.392673 18043 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 19:57:21.361465 18035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:22.829398 18037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:22.832383 18036 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1466 milliseconds
I20260924 19:57:22.832578 18027 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260924 19:57:22.833956 18027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:57:22.836997 18027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:57:22.838581 18027 hybrid_clock.cc:648] HybridClock initialized: now 1790279842838506 us; error 74 us; skew 500 ppm
I20260924 19:57:22.839270 18045 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:22.839843 18027 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:57:22.839781 18045 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:22.842870 17470 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:22.842859 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:22.844213 18045 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:22.845446 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:22.846166 18045 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 19:57:22.848569 18027 webserver.cc:533] Webserver started at http://127.17.2.131:37025/ using document root <none> and password file <none>
I20260924 19:57:22.849941 18027 fs_manager.cc:362] Metadata directory not provided
I20260924 19:57:22.850240 18027 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:57:22.850907 18027 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:57:22.857966 18027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/data/instance:
uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
format_stamp: "Formatted at 2026-09-24 19:57:22 on dist-test-slave-h376"
I20260924 19:57:22.859830 18027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/instance:
uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
format_stamp: "Formatted at 2026-09-24 19:57:22 on dist-test-slave-h376"
I20260924 19:57:22.870107 18027 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260924 19:57:22.878329 18049 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:22.879817 18027 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 19:57:22.880259 18027 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal
uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
format_stamp: "Formatted at 2026-09-24 19:57:22 on dist-test-slave-h376"
I20260924 19:57:22.880640 18027 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:57:22.957872 18027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:57:22.959527 18027 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:57:22.959997 18027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:57:22.963030 18027 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260924 19:57:22.965181 18055 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:22.965706 18055 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:22.968781 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:22.969280 18027 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:57:22.969553 18027 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:22.969884 18027 ts_tablet_manager.cc:616] Registered 0 tablets
I20260924 19:57:22.970135 18027 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 19:57:22.972934 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:22.974190 18055 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:22.974779 17446 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:22.975668 18055 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 19:57:23.161829 18027 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.131:40575
I20260924 19:57:23.161957 18162 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.131:40575 every 8 connection(s)
I20260924 19:57:23.164855 18027 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/data/info.pb
I20260924 19:57:23.170060 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 18027
I20260924 19:57:23.170677 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/instance
I20260924 19:57:23.245268 18164 heartbeater.cc:344] Connected to a master server at 127.17.2.189:43689
I20260924 19:57:23.245877 18164 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:23.247370 18164 heartbeater.cc:507] Master 127.17.2.189:43689 requested a full tablet report, sending...
I20260924 19:57:23.251276 17535 ts_manager.cc:194] Registered new tserver with Master: eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:23.257323 18163 heartbeater.cc:344] Connected to a master server at 127.17.2.188:38761
I20260924 19:57:23.257764 18163 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:23.258705 18163 heartbeater.cc:507] Master 127.17.2.188:38761 requested a full tablet report, sending...
I20260924 19:57:23.261065 17615 ts_manager.cc:194] Registered new tserver with Master: eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:23.263598 17615 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.17.2.131:47219
I20260924 19:57:23.281750 18166 heartbeater.cc:344] Connected to a master server at 127.17.2.190:46309
I20260924 19:57:23.282119 18166 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:23.282747 18166 heartbeater.cc:507] Master 127.17.2.190:46309 requested a full tablet report, sending...
I20260924 19:57:23.285960 17460 ts_manager.cc:194] Registered new tserver with Master: eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:23.296197 17418 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260924 19:57:23.364192 18180 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:23.364822 18180 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:23.367094 18180 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:23.368542 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:23.368953 17470 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:23.373268 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:23.374879 18180 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 19:57:23.379295 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_7ec33bafdfeb46d492e58bef6844de56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:23.384531 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_1e00c774eecf45f6952f8e381daeddbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:23.385628 17613 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_61920aedccc242398fa8dfaa779522e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:23.386736 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e00c774eecf45f6952f8e381daeddbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:23.388082 17613 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61920aedccc242398fa8dfaa779522e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:23.390478 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ec33bafdfeb46d492e58bef6844de56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:23.395589 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_21d6f3fb267e4fdfa178182aa817c45a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:23.397584 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21d6f3fb267e4fdfa178182aa817c45a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:23.469902 17418 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu with pid 17427
I20260924 19:57:23.516479 17614 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_1e00c774eecf45f6952f8e381daeddbd" } new_table_name: "default.Table_719c7b80244545adbf59004461c32c61" modify_external_catalogs: true
W20260924 19:57:23.519454 17602 connection.cc:573] client connection to 127.17.2.190:46309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:57:23.519591 17908 connection.cc:573] client connection to 127.17.2.190:46309 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:57:23.520012 17602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:57:23.521539 18019 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:46309 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:57:23.550673 17615 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_61920aedccc242398fa8dfaa779522e2" } new_table_name: "default.Table_09271c6b3dfb4a80a244f7241e1815ea" modify_external_catalogs: true
W20260924 19:57:23.559234 17602 consensus_peers.cc:597] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e -> Peer 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309): Couldn't send request to peer 3e059bbe4dac4178bccb3be48cb1118d. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:23.559474 17613 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_7ec33bafdfeb46d492e58bef6844de56" } modify_external_catalogs: true
I20260924 19:57:23.560230 17613 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_7ec33bafdfeb46d492e58bef6844de56" } modify_external_catalogs: true
I20260924 19:57:23.587981 17612 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_21d6f3fb267e4fdfa178182aa817c45a" } modify_external_catalogs: true
I20260924 19:57:23.588757 17612 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_21d6f3fb267e4fdfa178182aa817c45a" } modify_external_catalogs: true
W20260924 19:57:23.627935 17614 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0c86b120e4641a78b519821157f99d9 consensus state has no leader
W20260924 19:57:23.633195 17615 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c6943a365da464e83cedc2b9711ca89 has no consensus state
I20260924 19:57:23.646493 17613 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet e1931018e8134012a655a2a6e72f9a8d
I20260924 19:57:23.666778 17952 tablet_service.cc:1511] Processing CreateTablet for tablet e0c86b120e4641a78b519821157f99d9 (DEFAULT_TABLE table=default.Table_1e00c774eecf45f6952f8e381daeddbd [id=3fcc00879076471db1969ca4414ec8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:23.671648 17612 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet d960760e92b840cb9daf7ce08fce947d
I20260924 19:57:23.670675 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0c86b120e4641a78b519821157f99d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:23.689294 18098 tablet_service.cc:1511] Processing CreateTablet for tablet e0c86b120e4641a78b519821157f99d9 (DEFAULT_TABLE table=default.Table_1e00c774eecf45f6952f8e381daeddbd [id=3fcc00879076471db1969ca4414ec8f9]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:57:23.691277 17872 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:46309 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111)
W20260924 19:57:23.698966 17600 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_09271c6b3dfb4a80a244f7241e1815ea [id=5878a9a1b1d54055a5e8dc712b9de665], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0c6943a365da464e83cedc2b9711ca89 consensus state has no leader
I20260924 19:57:23.701133 17802 tablet_service.cc:1511] Processing CreateTablet for tablet e0c86b120e4641a78b519821157f99d9 (DEFAULT_TABLE table=default.Table_1e00c774eecf45f6952f8e381daeddbd [id=3fcc00879076471db1969ca4414ec8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:23.703277 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0c86b120e4641a78b519821157f99d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:23.705855 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0c86b120e4641a78b519821157f99d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:23.709049 17951 tablet_service.cc:1511] Processing CreateTablet for tablet 0c6943a365da464e83cedc2b9711ca89 (DEFAULT_TABLE table=default.Table_09271c6b3dfb4a80a244f7241e1815ea [id=5878a9a1b1d54055a5e8dc712b9de665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:23.710774 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c6943a365da464e83cedc2b9711ca89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:23.712929 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 0c6943a365da464e83cedc2b9711ca89 (DEFAULT_TABLE table=default.Table_09271c6b3dfb4a80a244f7241e1815ea [id=5878a9a1b1d54055a5e8dc712b9de665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:23.714730 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c6943a365da464e83cedc2b9711ca89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:23.721052 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_36206632626543c09bf095542225bd94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:23.724799 18220 tablet_bootstrap.cc:492] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:23.725222 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 0c6943a365da464e83cedc2b9711ca89 (DEFAULT_TABLE table=default.Table_09271c6b3dfb4a80a244f7241e1815ea [id=5878a9a1b1d54055a5e8dc712b9de665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:23.727025 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c6943a365da464e83cedc2b9711ca89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:23.744036 18220 tablet_bootstrap.cc:654] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:23.746737 18220 log.cc:826] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:57:23.796363 18163 heartbeater.cc:499] Master 127.17.2.188:38761 was elected leader, sending a full tablet report...
I20260924 19:57:23.824010 17613 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_ed65d70c9c8046b384abc59fccca5771"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:23.825832 17613 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed65d70c9c8046b384abc59fccca5771 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:23.827647 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_d2abf2d7d4494abc8274e2ce9888b9f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:23.829355 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2abf2d7d4494abc8274e2ce9888b9f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:23.830401 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_b88ea8b3f5ed42cd94da21ec1f56aaca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:23.832060 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b88ea8b3f5ed42cd94da21ec1f56aaca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:23.834743 18166 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:46309 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111)
I20260924 19:57:23.838130 17611 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_09271c6b3dfb4a80a244f7241e1815ea" } new_table_name: "default.Table_69b957785d9d4791aa08d3755602bb87" modify_external_catalogs: true
I20260924 19:57:23.840754 17610 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_719c7b80244545adbf59004461c32c61" } new_table_name: "default.Table_023f1818503b483f9b6bbcabc142d830" modify_external_catalogs: true
I20260924 19:57:23.856164 18224 tablet_bootstrap.cc:492] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:23.867043 18224 tablet_bootstrap.cc:654] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:23.869539 18224 log.cc:826] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202: Log is configured to *not* fsync() on all Append() calls
W20260924 19:57:23.884857 17610 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0c86b120e4641a78b519821157f99d9 consensus state has no leader
W20260924 19:57:23.893087 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36206632626543c09bf095542225bd94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:23.908113 17609 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b88ea8b3f5ed42cd94da21ec1f56aaca" } modify_external_catalogs: true
I20260924 19:57:23.908877 17609 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b88ea8b3f5ed42cd94da21ec1f56aaca" } modify_external_catalogs: true
I20260924 19:57:23.911985 17610 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_023f1818503b483f9b6bbcabc142d830" } modify_external_catalogs: true
I20260924 19:57:23.912693 17610 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_023f1818503b483f9b6bbcabc142d830" } modify_external_catalogs: true
I20260924 19:57:23.919876 18220 tablet_bootstrap.cc:492] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:23.920461 18220 ts_tablet_manager.cc:1403] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.196s user 0.041s sys 0.030s
I20260924 19:57:23.931414 18224 tablet_bootstrap.cc:492] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:23.932277 18224 ts_tablet_manager.cc:1403] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.077s user 0.030s sys 0.022s
I20260924 19:57:23.932533 18223 tablet_bootstrap.cc:492] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:23.933418 17611 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c6943a365da464e83cedc2b9711ca89 consensus state has no leader
I20260924 19:57:23.940548 18223 tablet_bootstrap.cc:654] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:23.942934 18223 log.cc:826] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: Log is configured to *not* fsync() on all Append() calls
I20260924 19:57:23.961691 18223 tablet_bootstrap.cc:492] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:23.962412 18223 ts_tablet_manager.cc:1403] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.000s
I20260924 19:57:23.979360 18220 raft_consensus.cc:359] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:23.983457 18220 raft_consensus.cc:385] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:23.983886 18220 raft_consensus.cc:740] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:23.985186 18220 consensus_queue.cc:260] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:23.999146 18220 ts_tablet_manager.cc:1434] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.078s user 0.027s sys 0.009s
I20260924 19:57:24.013538 18220 tablet_bootstrap.cc:492] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:24.028574 18227 raft_consensus.cc:493] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:24.029316 18227 raft_consensus.cc:515] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:24.050166 17609 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 001cc96272d245adbf79fff66723d55c
I20260924 19:57:24.051656 18227 leader_election.cc:290] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:24.051048 18220 tablet_bootstrap.cc:654] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.079761 18224 raft_consensus.cc:359] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:24.081422 18224 raft_consensus.cc:385] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.082079 17610 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet e0c86b120e4641a78b519821157f99d9
I20260924 19:57:24.081957 18224 raft_consensus.cc:740] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.082592 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0c86b120e4641a78b519821157f99d9" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:24.083431 18224 consensus_queue.cc:260] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:24.086444 17802 tablet_service.cc:1558] Processing DeleteTablet for tablet e0c86b120e4641a78b519821157f99d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:23 UTC) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:24.089722 17905 leader_election.cc:343] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:24.095569 17951 tablet_service.cc:1558] Processing DeleteTablet for tablet e0c86b120e4641a78b519821157f99d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:23 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:24.099952 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet e0c86b120e4641a78b519821157f99d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:23 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:24.096446 18223 raft_consensus.cc:359] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:24.102094 18223 raft_consensus.cc:385] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.102423 18223 raft_consensus.cc:740] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.110492 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0c86b120e4641a78b519821157f99d9" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:24.111439 17823 raft_consensus.cc:2468] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:24.113008 17906 leader_election.cc:304] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters: eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:24.113056 18223 consensus_queue.cc:260] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:24.114570 18227 raft_consensus.cc:2804] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:24.114989 18227 raft_consensus.cc:493] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:24.115336 18227 raft_consensus.cc:3060] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:24.118407 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet e0c86b120e4641a78b519821157f99d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0c86b120e4641a78b519821157f99d9 already in progress: creating tablet
I20260924 19:57:24.119293 18220 tablet_bootstrap.cc:492] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:24.120023 18220 ts_tablet_manager.cc:1403] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.107s user 0.012s sys 0.003s
I20260924 19:57:24.123343 18220 raft_consensus.cc:359] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:24.124084 18220 raft_consensus.cc:385] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.124411 18220 raft_consensus.cc:740] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.125463 18220 consensus_queue.cc:260] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:24.129235 18212 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:24.135030 18212 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:24.140957 18220 ts_tablet_manager.cc:1434] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.021s user 0.006s sys 0.004s
I20260924 19:57:24.134023 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_105481a12d7643f08276f7a5dcedf484"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:24.151364 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_105481a12d7643f08276f7a5dcedf484 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:24.154592 17602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.190:46309, user_credentials={real_user=slave}} blocked reactor thread for 49736.1us
I20260924 19:57:24.141921 18224 ts_tablet_manager.cc:1434] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.209s user 0.049s sys 0.004s
I20260924 19:57:24.164816 17610 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_ed65d70c9c8046b384abc59fccca5771" } new_table_name: "default.Table_4a11f058715140cba5789985f21767e0" modify_external_catalogs: true
I20260924 19:57:24.165040 17613 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_36206632626543c09bf095542225bd94" } new_table_name: "default.Table_0704f9e805c94a08b7c29d68d253334b" modify_external_catalogs: true
W20260924 19:57:24.168834 18025 tablet.cc:2433] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:57:24.169106 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet e0c86b120e4641a78b519821157f99d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0c86b120e4641a78b519821157f99d9 already in progress: creating tablet
I20260924 19:57:24.175290 17802 tablet_service.cc:1558] Processing DeleteTablet for tablet e0c86b120e4641a78b519821157f99d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:23 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:24.176306 18231 tablet_replica.cc:333] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:24.177120 18231 raft_consensus.cc:2243] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:24.177722 18231 raft_consensus.cc:2272] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:24.181365 18231 ts_tablet_manager.cc:1916] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:24.124989 18227 raft_consensus.cc:515] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:24.196002 18227 leader_election.cc:290] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:24.197090 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_bdbe025688bd462abbcacc1b6b3fdf33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:24.198451 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0c86b120e4641a78b519821157f99d9" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:24.198414 18232 tablet_replica.cc:333] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:24.199043 18118 raft_consensus.cc:3060] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:24.199703 18232 raft_consensus.cc:2243] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:24.200287 18232 raft_consensus.cc:2272] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:24.200278 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdbe025688bd462abbcacc1b6b3fdf33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:24.203387 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0c86b120e4641a78b519821157f99d9" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
W20260924 19:57:24.206125 17906 leader_election.cc:343] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: Tablet not RUNNING: STOPPED
I20260924 19:57:24.206831 18223 ts_tablet_manager.cc:1434] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.244s user 0.043s sys 0.010s
I20260924 19:57:24.208160 18223 tablet_bootstrap.cc:492] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:24.208421 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_25917a01f36140dfa4f504bc59aaf32b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:24.210148 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25917a01f36140dfa4f504bc59aaf32b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:24.213109 18232 ts_tablet_manager.cc:1916] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:24.224109 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_88bf80a74c3245d1a74b3757ef4eae71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:24.225998 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88bf80a74c3245d1a74b3757ef4eae71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:24.226070 18224 tablet_bootstrap.cc:492] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:24.233753 18170 tablet.cc:2433] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:57:24.235368 17613 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d4bee1eaae04ce99c7684c3da321c36 has no consensus state
W20260924 19:57:24.236793 17610 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6f7f650fc26498faff2059b42bbf91f has no consensus state
I20260924 19:57:24.237567 18223 tablet_bootstrap.cc:654] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.245862 18224 tablet_bootstrap.cc:654] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.248385 18118 raft_consensus.cc:2468] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:24.257620 17610 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_d2abf2d7d4494abc8274e2ce9888b9f3" } modify_external_catalogs: true
I20260924 19:57:24.257714 18231 ts_tablet_manager.cc:1929] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:24.258304 17610 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_d2abf2d7d4494abc8274e2ce9888b9f3" } modify_external_catalogs: true
I20260924 19:57:24.261178 17905 leader_election.cc:304] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
W20260924 19:57:24.261912 17905 raft_consensus.cc:2686] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260924 19:57:24.262894 18231 log.cc:1199] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/e0c86b120e4641a78b519821157f99d9
I20260924 19:57:24.264029 18231 ts_tablet_manager.cc:1950] T e0c86b120e4641a78b519821157f99d9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:24.264564 18232 ts_tablet_manager.cc:1929] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:24.265138 18232 log.cc:1199] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/e0c86b120e4641a78b519821157f99d9
I20260924 19:57:24.266112 18232 ts_tablet_manager.cc:1950] T e0c86b120e4641a78b519821157f99d9 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:24.270628 18223 tablet_bootstrap.cc:492] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:24.271080 18223 ts_tablet_manager.cc:1403] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.009s
I20260924 19:57:24.274498 18223 raft_consensus.cc:359] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:24.275406 18223 raft_consensus.cc:385] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.275751 18223 raft_consensus.cc:740] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.276674 18223 consensus_queue.cc:260] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:24.278191 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet e0c86b120e4641a78b519821157f99d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:23 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:24.282207 18224 tablet_bootstrap.cc:492] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:24.282259 18223 ts_tablet_manager.cc:1434] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20260924 19:57:24.282729 18224 ts_tablet_manager.cc:1403] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.057s user 0.013s sys 0.004s
I20260924 19:57:24.284606 18233 tablet_replica.cc:333] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:24.288106 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet e0c86b120e4641a78b519821157f99d9 (table default.Table_023f1818503b483f9b6bbcabc142d830 [id=3fcc00879076471db1969ca4414ec8f9]) successfully deleted
I20260924 19:57:24.288173 18224 raft_consensus.cc:359] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:24.288693 18233 raft_consensus.cc:2243] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:24.288863 18224 raft_consensus.cc:385] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.289130 18224 raft_consensus.cc:740] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.289163 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet e0c86b120e4641a78b519821157f99d9 (table default.Table_023f1818503b483f9b6bbcabc142d830 [id=3fcc00879076471db1969ca4414ec8f9]) successfully deleted
I20260924 19:57:24.289283 18233 raft_consensus.cc:2272] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:24.289860 18224 consensus_queue.cc:260] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:24.291481 17612 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_105481a12d7643f08276f7a5dcedf484" } modify_external_catalogs: true
I20260924 19:57:24.292184 18233 ts_tablet_manager.cc:1916] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:24.292047 17612 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_105481a12d7643f08276f7a5dcedf484" } modify_external_catalogs: true
I20260924 19:57:24.308411 18224 ts_tablet_manager.cc:1434] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.025s user 0.003s sys 0.004s
I20260924 19:57:24.321280 18233 ts_tablet_manager.cc:1929] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:24.334964 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_69b957785d9d4791aa08d3755602bb87 [id=5878a9a1b1d54055a5e8dc712b9de665]). Expected version 2 got 0
I20260924 19:57:24.338611 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_69b957785d9d4791aa08d3755602bb87 [id=5878a9a1b1d54055a5e8dc712b9de665]). Expected version 2 got 0
I20260924 19:57:24.342367 17802 tablet_service.cc:1511] Processing CreateTablet for tablet c6f7f650fc26498faff2059b42bbf91f (DEFAULT_TABLE table=default.Table_4a11f058715140cba5789985f21767e0 [id=c3b94a6ee3b5400cb352433d44a902f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.343511 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f7f650fc26498faff2059b42bbf91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.347033 17613 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_69b957785d9d4791aa08d3755602bb87 [id=5878a9a1b1d54055a5e8dc712b9de665]). Expected version 2 got 0
I20260924 19:57:24.354647 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 7d4bee1eaae04ce99c7684c3da321c36 (DEFAULT_TABLE table=default.Table_0704f9e805c94a08b7c29d68d253334b [id=965f3deebb8c4029901624f08a79febc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.355834 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4bee1eaae04ce99c7684c3da321c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.356690 17607 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_69b957785d9d4791aa08d3755602bb87 [id=5878a9a1b1d54055a5e8dc712b9de665]). Expected version 2 got 0
I20260924 19:57:24.361130 17951 tablet_service.cc:1511] Processing CreateTablet for tablet c6f7f650fc26498faff2059b42bbf91f (DEFAULT_TABLE table=default.Table_4a11f058715140cba5789985f21767e0 [id=c3b94a6ee3b5400cb352433d44a902f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.361347 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_69b957785d9d4791aa08d3755602bb87 [id=5878a9a1b1d54055a5e8dc712b9de665]). Expected version 2 got 0
I20260924 19:57:24.362242 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f7f650fc26498faff2059b42bbf91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.364763 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_69b957785d9d4791aa08d3755602bb87 [id=5878a9a1b1d54055a5e8dc712b9de665]). Expected version 2 got 0
I20260924 19:57:24.373137 17800 tablet_service.cc:1511] Processing CreateTablet for tablet d73260e74a324081bda4457a9e9931c9 (DEFAULT_TABLE table=default.Table_d2abf2d7d4494abc8274e2ce9888b9f3 [id=e025f80a4e9744c5ba260b228f5e79ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.374491 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d73260e74a324081bda4457a9e9931c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:24.375540 17878 tablet.cc:2433] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:57:24.382247 17952 tablet_service.cc:1511] Processing CreateTablet for tablet 7d4bee1eaae04ce99c7684c3da321c36 (DEFAULT_TABLE table=default.Table_0704f9e805c94a08b7c29d68d253334b [id=965f3deebb8c4029901624f08a79febc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.383502 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4bee1eaae04ce99c7684c3da321c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.338042 18098 tablet_service.cc:1511] Processing CreateTablet for tablet c6f7f650fc26498faff2059b42bbf91f (DEFAULT_TABLE table=default.Table_4a11f058715140cba5789985f21767e0 [id=c3b94a6ee3b5400cb352433d44a902f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.398154 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f7f650fc26498faff2059b42bbf91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.381757 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 7d4bee1eaae04ce99c7684c3da321c36 (DEFAULT_TABLE table=default.Table_0704f9e805c94a08b7c29d68d253334b [id=965f3deebb8c4029901624f08a79febc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.403950 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d4bee1eaae04ce99c7684c3da321c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.412297 18220 tablet_bootstrap.cc:492] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:24.416064 18096 tablet_service.cc:1511] Processing CreateTablet for tablet d73260e74a324081bda4457a9e9931c9 (DEFAULT_TABLE table=default.Table_d2abf2d7d4494abc8274e2ce9888b9f3 [id=e025f80a4e9744c5ba260b228f5e79ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.417297 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d73260e74a324081bda4457a9e9931c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.419028 18220 tablet_bootstrap.cc:654] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.431421 18233 log.cc:1199] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/e0c86b120e4641a78b519821157f99d9
I20260924 19:57:24.432783 18233 ts_tablet_manager.cc:1950] T e0c86b120e4641a78b519821157f99d9 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:24.436942 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet e0c86b120e4641a78b519821157f99d9 (table default.Table_023f1818503b483f9b6bbcabc142d830 [id=3fcc00879076471db1969ca4414ec8f9]) successfully deleted
I20260924 19:57:24.438622 18220 tablet_bootstrap.cc:492] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:24.439205 18220 ts_tablet_manager.cc:1403] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.005s
I20260924 19:57:24.442011 18220 raft_consensus.cc:359] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.442849 18220 raft_consensus.cc:385] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.443213 18220 raft_consensus.cc:740] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.446637 17614 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_0704f9e805c94a08b7c29d68d253334b" } new_table_name: "default.Table_b06e7b3533ae4657991ac1c424cc08cd" modify_external_catalogs: true
I20260924 19:57:24.448558 18224 tablet_bootstrap.cc:492] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:24.455492 18220 consensus_queue.cc:260] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.458205 18220 ts_tablet_manager.cc:1434] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20260924 19:57:24.458720 18224 tablet_bootstrap.cc:654] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.459348 18220 tablet_bootstrap.cc:492] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:24.466106 18220 tablet_bootstrap.cc:654] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.469722 17952 tablet_service.cc:1511] Processing CreateTablet for tablet d73260e74a324081bda4457a9e9931c9 (DEFAULT_TABLE table=default.Table_d2abf2d7d4494abc8274e2ce9888b9f3 [id=e025f80a4e9744c5ba260b228f5e79ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.470963 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d73260e74a324081bda4457a9e9931c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.473858 17611 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_bdbe025688bd462abbcacc1b6b3fdf33" } new_table_name: "default.Table_810632cce6a446c5adb1739f75853ce8" modify_external_catalogs: true
I20260924 19:57:24.479415 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 7d4bee1eaae04ce99c7684c3da321c36 (table default.Table_0704f9e805c94a08b7c29d68d253334b [id=965f3deebb8c4029901624f08a79febc]). Expected version 1 got 0
I20260924 19:57:24.488415 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_d3ef2c88a2bf4ccb9f679aa75ebe0309"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:24.490264 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3ef2c88a2bf4ccb9f679aa75ebe0309 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:24.507354 18223 tablet_bootstrap.cc:492] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:24.511878 17612 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 73a13e752613410cbe0a1c2d47c2d074
I20260924 19:57:24.514030 18223 tablet_bootstrap.cc:654] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.517349 18220 tablet_bootstrap.cc:492] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:24.518062 18220 ts_tablet_manager.cc:1403] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.005s
I20260924 19:57:24.531769 18220 raft_consensus.cc:359] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.534314 18220 raft_consensus.cc:385] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.534729 18220 raft_consensus.cc:740] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.543571 18220 consensus_queue.cc:260] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.544582 17610 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet d73260e74a324081bda4457a9e9931c9
I20260924 19:57:24.546764 18220 ts_tablet_manager.cc:1434] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260924 19:57:24.547941 18220 tablet_bootstrap.cc:492] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:24.548671 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet d73260e74a324081bda4457a9e9931c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:24.550525 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet d73260e74a324081bda4457a9e9931c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d73260e74a324081bda4457a9e9931c9 already in progress: creating tablet
I20260924 19:57:24.554935 18220 tablet_bootstrap.cc:654] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.564504 18096 tablet_service.cc:1558] Processing DeleteTablet for tablet d73260e74a324081bda4457a9e9931c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:24.566534 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet d73260e74a324081bda4457a9e9931c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d73260e74a324081bda4457a9e9931c9 already in progress: creating tablet
I20260924 19:57:24.569052 18223 tablet_bootstrap.cc:492] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:24.569491 18223 ts_tablet_manager.cc:1403] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.062s user 0.017s sys 0.009s
I20260924 19:57:24.572419 18223 raft_consensus.cc:359] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.573215 18223 raft_consensus.cc:385] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.573546 18223 raft_consensus.cc:740] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.574507 18224 tablet_bootstrap.cc:492] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:24.574533 18223 consensus_queue.cc:260] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.565398 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet d73260e74a324081bda4457a9e9931c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:24.575181 18224 ts_tablet_manager.cc:1403] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.128s user 0.033s sys 0.026s
W20260924 19:57:24.577070 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet d73260e74a324081bda4457a9e9931c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d73260e74a324081bda4457a9e9931c9 already in progress: creating tablet
I20260924 19:57:24.578011 18224 raft_consensus.cc:359] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.578682 18224 raft_consensus.cc:385] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.578953 18224 raft_consensus.cc:740] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.579952 18224 consensus_queue.cc:260] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.584993 18224 ts_tablet_manager.cc:1434] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260924 19:57:24.586091 18223 ts_tablet_manager.cc:1434] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260924 19:57:24.591343 18223 tablet_bootstrap.cc:492] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:24.597826 18224 tablet_bootstrap.cc:492] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:24.598614 18223 tablet_bootstrap.cc:654] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.604260 18224 tablet_bootstrap.cc:654] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.612653 18223 tablet_bootstrap.cc:492] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:24.613154 18223 ts_tablet_manager.cc:1403] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.000s
I20260924 19:57:24.616282 18223 raft_consensus.cc:359] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.617018 18223 raft_consensus.cc:385] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.617494 18223 raft_consensus.cc:740] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.618413 18223 consensus_queue.cc:260] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.627857 17609 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_b200a0c5e5014cc09c18338e29eb3e2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:24.628950 18223 ts_tablet_manager.cc:1434] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
W20260924 19:57:24.630450 17609 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b200a0c5e5014cc09c18338e29eb3e2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:24.632227 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_87b845fa3a064c269030287021ae1c05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:24.634079 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87b845fa3a064c269030287021ae1c05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:24.636356 18223 tablet_bootstrap.cc:492] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:24.638631 18224 tablet_bootstrap.cc:492] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:24.639353 18224 ts_tablet_manager.cc:1403] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.003s
I20260924 19:57:24.639964 18235 raft_consensus.cc:493] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:24.640475 18235 raft_consensus.cc:515] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:24.643988 17611 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0fbce82a1ae49409de07104f2da8c8f has no consensus state
I20260924 19:57:24.651361 18236 raft_consensus.cc:493] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:24.651821 18236 raft_consensus.cc:515] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:24.654937 18236 leader_election.cc:290] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:24.654932 18227 raft_consensus.cc:493] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:24.655642 18227 raft_consensus.cc:515] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.656298 18224 raft_consensus.cc:359] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.657181 18224 raft_consensus.cc:385] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.657533 18224 raft_consensus.cc:740] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.657624 18227 leader_election.cc:290] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:24.658564 18224 consensus_queue.cc:260] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.660153 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6f7f650fc26498faff2059b42bbf91f" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:24.660758 17823 raft_consensus.cc:2468] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:24.661757 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6f7f650fc26498faff2059b42bbf91f" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:24.662352 18118 raft_consensus.cc:2468] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:24.665323 17905 leader_election.cc:304] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:24.669044 18224 ts_tablet_manager.cc:1434] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.029s user 0.002s sys 0.005s
I20260924 19:57:24.669632 18227 raft_consensus.cc:2804] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:24.669973 18224 tablet_bootstrap.cc:492] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:24.670136 18227 raft_consensus.cc:493] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:24.670637 18227 raft_consensus.cc:3060] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:24.680768 18227 raft_consensus.cc:515] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.682145 18235 leader_election.cc:290] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:24.685524 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6f7f650fc26498faff2059b42bbf91f" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:24.686178 17823 raft_consensus.cc:3060] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:24.693025 18224 tablet_bootstrap.cc:654] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.698750 18223 tablet_bootstrap.cc:654] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:24.697335 17823 raft_consensus.cc:2468] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:24.702673 17906 leader_election.cc:304] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:24.703832 18220 tablet_bootstrap.cc:492] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:24.704341 18220 ts_tablet_manager.cc:1403] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.157s user 0.021s sys 0.028s
I20260924 19:57:24.706300 18227 leader_election.cc:290] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:24.701248 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6f7f650fc26498faff2059b42bbf91f" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:24.707101 18118 raft_consensus.cc:3060] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:24.707644 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_0954c32d949b48bebae0462522b8da72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:24.709443 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0954c32d949b48bebae0462522b8da72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:24.713949 18118 raft_consensus.cc:2468] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:24.715620 17613 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_810632cce6a446c5adb1739f75853ce8" } modify_external_catalogs: true
I20260924 19:57:24.707312 18220 raft_consensus.cc:359] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.716234 17613 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_810632cce6a446c5adb1739f75853ce8" } modify_external_catalogs: true
I20260924 19:57:24.716437 18220 raft_consensus.cc:385] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.716769 18220 raft_consensus.cc:740] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.717531 18220 consensus_queue.cc:260] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.716933 18227 raft_consensus.cc:2804] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:24.718243 18224 tablet_bootstrap.cc:492] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:24.718662 18224 ts_tablet_manager.cc:1403] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.007s
I20260924 19:57:24.719786 18220 ts_tablet_manager.cc:1434] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260924 19:57:24.722061 18224 raft_consensus.cc:359] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.723091 18224 raft_consensus.cc:385] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.723973 18224 raft_consensus.cc:740] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.726118 18227 raft_consensus.cc:697] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:24.729096 18227 consensus_queue.cc:237] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.732200 18224 consensus_queue.cc:260] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.734714 18224 ts_tablet_manager.cc:1434] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.016s user 0.004s sys 0.004s
I20260924 19:57:24.742482 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 7d4bee1eaae04ce99c7684c3da321c36 (table default.Table_0704f9e805c94a08b7c29d68d253334b [id=965f3deebb8c4029901624f08a79febc]). Expected version 1 got 0
W20260924 19:57:24.743829 17614 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d4bee1eaae04ce99c7684c3da321c36 consensus state has no leader
I20260924 19:57:24.747812 17612 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet c6f7f650fc26498faff2059b42bbf91f (table default.Table_4a11f058715140cba5789985f21767e0 [id=c3b94a6ee3b5400cb352433d44a902f4]). Expected version 1 got 0
W20260924 19:57:24.763944 17599 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet e0fbce82a1ae49409de07104f2da8c8f (table default.Table_810632cce6a446c5adb1739f75853ce8 [id=1105edf1f289484c948e5568c810cdd3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e0fbce82a1ae49409de07104f2da8c8f consensus state has no leader
I20260924 19:57:24.768921 17612 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_d3ef2c88a2bf4ccb9f679aa75ebe0309" } modify_external_catalogs: true
I20260924 19:57:24.769484 17612 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_d3ef2c88a2bf4ccb9f679aa75ebe0309" } modify_external_catalogs: true
I20260924 19:57:24.769941 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 32466931871c4c5d8e9f95a11ffe26be (DEFAULT_TABLE table=default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.771086 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32466931871c4c5d8e9f95a11ffe26be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.774197 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet c6f7f650fc26498faff2059b42bbf91f (table default.Table_4a11f058715140cba5789985f21767e0 [id=c3b94a6ee3b5400cb352433d44a902f4]). Expected version 1 got 0
I20260924 19:57:24.775936 18235 raft_consensus.cc:493] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:24.776391 18235 raft_consensus.cc:515] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.774951 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 7d4bee1eaae04ce99c7684c3da321c36 (table default.Table_b06e7b3533ae4657991ac1c424cc08cd [id=965f3deebb8c4029901624f08a79febc]). Expected version 2 got 0
I20260924 19:57:24.778935 18235 leader_election.cc:290] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:24.782250 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet d73260e74a324081bda4457a9e9931c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:24.783092 18231 tablet_replica.cc:333] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
W20260924 19:57:24.783633 17646 debug-util.cc:398] Leaking SignalData structure 0x7b08000420a0 after lost signal to thread 17579
I20260924 19:57:24.783914 18231 raft_consensus.cc:2243] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260924 19:57:24.784329 17646 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9f40 after lost signal to thread 17649
I20260924 19:57:24.784353 18231 raft_consensus.cc:2272] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:24.786856 18231 ts_tablet_manager.cc:1916] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:24.783093 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 7d4bee1eaae04ce99c7684c3da321c36 (table default.Table_b06e7b3533ae4657991ac1c424cc08cd [id=965f3deebb8c4029901624f08a79febc]). Expected version 2 got 0
I20260924 19:57:24.791560 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet c6f7f650fc26498faff2059b42bbf91f (table default.Table_4a11f058715140cba5789985f21767e0 [id=c3b94a6ee3b5400cb352433d44a902f4]). Expected version 1 got 0
I20260924 19:57:24.804540 18096 tablet_service.cc:1511] Processing CreateTablet for tablet 32466931871c4c5d8e9f95a11ffe26be (DEFAULT_TABLE table=default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.805663 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32466931871c4c5d8e9f95a11ffe26be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.813483 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6943a365da464e83cedc2b9711ca89" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:24.814373 17972 raft_consensus.cc:2468] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:24.816619 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6943a365da464e83cedc2b9711ca89" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:24.817237 17823 raft_consensus.cc:2468] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:24.818508 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d4bee1eaae04ce99c7684c3da321c36" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:24.819046 17823 raft_consensus.cc:2468] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:24.820369 18051 leader_election.cc:304] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:24.821331 18235 raft_consensus.cc:2804] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:24.822499 18235 raft_consensus.cc:493] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:24.822840 18235 raft_consensus.cc:3060] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:24.826056 18223 tablet_bootstrap.cc:492] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:24.833770 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6943a365da464e83cedc2b9711ca89" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:24.834390 17972 raft_consensus.cc:2468] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:24.834820 17952 tablet_service.cc:1511] Processing CreateTablet for tablet 32466931871c4c5d8e9f95a11ffe26be (DEFAULT_TABLE table=default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.835951 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32466931871c4c5d8e9f95a11ffe26be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.841267 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d4bee1eaae04ce99c7684c3da321c36" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:24.841864 17972 raft_consensus.cc:2468] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:24.845389 17753 leader_election.cc:304] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:24.846489 18236 raft_consensus.cc:2804] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:24.847029 18236 raft_consensus.cc:493] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:24.847496 18236 raft_consensus.cc:3060] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:24.829325 18235 raft_consensus.cc:515] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:24.830483 18051 leader_election.cc:304] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:24.863425 18235 leader_election.cc:290] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:24.864048 18235 raft_consensus.cc:2804] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:24.864403 18235 raft_consensus.cc:493] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:24.864753 18235 raft_consensus.cc:3060] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:24.866034 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6943a365da464e83cedc2b9711ca89" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:24.866586 17972 raft_consensus.cc:3060] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:24.843991 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6943a365da464e83cedc2b9711ca89" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:24.871675 18118 raft_consensus.cc:2393] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c4f6e3a7b06148918d230c77ab0c8202 in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:24.872216 18223 ts_tablet_manager.cc:1403] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.236s user 0.009s sys 0.003s
I20260924 19:57:24.862372 18236 raft_consensus.cc:515] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:24.877144 17972 raft_consensus.cc:2468] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:24.879354 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6943a365da464e83cedc2b9711ca89" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:24.880331 17971 raft_consensus.cc:2393] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c4f6e3a7b06148918d230c77ab0c8202 in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:24.880568 18236 leader_election.cc:290] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:24.883322 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 6edb7999719e4c3c93b93539bfe1550c (DEFAULT_TABLE table=default.Table_88bf80a74c3245d1a74b3757ef4eae71 [id=ecd769593d2d4b86a333ebf13c174a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.884382 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edb7999719e4c3c93b93539bfe1550c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.878468 18050 leader_election.cc:304] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:24.888268 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6943a365da464e83cedc2b9711ca89" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:24.888957 18118 raft_consensus.cc:2393] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c4f6e3a7b06148918d230c77ab0c8202 in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:24.891410 17753 leader_election.cc:304] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:24.892261 18236 raft_consensus.cc:2749] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 19:57:24.876487 18235 raft_consensus.cc:515] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.906148 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d4bee1eaae04ce99c7684c3da321c36" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:24.906749 17972 raft_consensus.cc:3060] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:24.911945 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 6edb7999719e4c3c93b93539bfe1550c (DEFAULT_TABLE table=default.Table_88bf80a74c3245d1a74b3757ef4eae71 [id=ecd769593d2d4b86a333ebf13c174a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.913122 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edb7999719e4c3c93b93539bfe1550c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.914247 18223 raft_consensus.cc:359] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:24.914929 18223 raft_consensus.cc:385] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:24.925779 18223 raft_consensus.cc:740] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:24.926482 18223 consensus_queue.cc:260] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
W20260924 19:57:24.925637 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 95205.2us
I20260924 19:57:24.935853 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6943a365da464e83cedc2b9711ca89" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:24.936693 17823 raft_consensus.cc:2393] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eaf64bfb64aa42948c9bc1411ce4ce35 in current term 1: Already voted for candidate c4f6e3a7b06148918d230c77ab0c8202 in this term.
I20260924 19:57:24.939082 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d4bee1eaae04ce99c7684c3da321c36" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:24.939620 17823 raft_consensus.cc:3060] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:24.944979 17951 tablet_service.cc:1511] Processing CreateTablet for tablet 6edb7999719e4c3c93b93539bfe1550c (DEFAULT_TABLE table=default.Table_88bf80a74c3245d1a74b3757ef4eae71 [id=ecd769593d2d4b86a333ebf13c174a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:24.946102 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edb7999719e4c3c93b93539bfe1550c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:24.948693 17609 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_4a11f058715140cba5789985f21767e0" } new_table_name: "default.Table_f570a48d43e84d5d86ef5a1a92e3161c" modify_external_catalogs: true
I20260924 19:57:24.952605 17610 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_e251c4f5fec44bc0952684c5ff3d826f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:24.953198 17608 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_69b957785d9d4791aa08d3755602bb87" } new_table_name: "default.Table_93373f379bb14b6ca7a3ac7a628cad63" modify_external_catalogs: true
W20260924 19:57:24.954315 17610 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e251c4f5fec44bc0952684c5ff3d826f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:24.958132 17823 raft_consensus.cc:2468] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:24.927752 18261 raft_consensus.cc:2804] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:24.933688 18235 leader_election.cc:290] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:24.983952 18261 raft_consensus.cc:697] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:24.984974 18261 consensus_queue.cc:237] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:24.986215 17972 raft_consensus.cc:2468] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:24.992340 18050 leader_election.cc:304] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:24.996176 18235 raft_consensus.cc:2804] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:25.000048 18235 raft_consensus.cc:697] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:25.001014 18235 consensus_queue.cc:237] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.009408 18220 tablet_bootstrap.cc:492] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:25.013826 17607 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 7d4bee1eaae04ce99c7684c3da321c36 (table default.Table_b06e7b3533ae4657991ac1c424cc08cd [id=965f3deebb8c4029901624f08a79febc]). Expected version 2 got 0
I20260924 19:57:25.016075 18220 tablet_bootstrap.cc:654] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:25.018766 17607 catalog_manager.cc:5838] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:25.022850 17607 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet c6f7f650fc26498faff2059b42bbf91f (table default.Table_4a11f058715140cba5789985f21767e0 [id=c3b94a6ee3b5400cb352433d44a902f4]). Expected version 1 got 0
I20260924 19:57:25.033528 18220 tablet_bootstrap.cc:492] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:25.033995 18220 ts_tablet_manager.cc:1403] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.003s
I20260924 19:57:25.037417 18220 raft_consensus.cc:359] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.038116 18220 raft_consensus.cc:385] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:25.038434 18220 raft_consensus.cc:740] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:25.043433 18220 consensus_queue.cc:260] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.044130 17952 tablet_service.cc:1511] Processing CreateTablet for tablet e0fbce82a1ae49409de07104f2da8c8f (DEFAULT_TABLE table=default.Table_810632cce6a446c5adb1739f75853ce8 [id=1105edf1f289484c948e5568c810cdd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:25.045181 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0fbce82a1ae49409de07104f2da8c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:25.046310 18220 ts_tablet_manager.cc:1434] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260924 19:57:25.047200 18220 tablet_bootstrap.cc:492] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:25.061034 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_a07eb85445714c59bb9837bc7228144c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:25.064354 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a07eb85445714c59bb9837bc7228144c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:25.066000 18220 tablet_bootstrap.cc:654] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:25.079057 18227 raft_consensus.cc:493] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:25.079553 18227 raft_consensus.cc:515] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.074179 18097 tablet_service.cc:1511] Processing CreateTablet for tablet e0fbce82a1ae49409de07104f2da8c8f (DEFAULT_TABLE table=default.Table_810632cce6a446c5adb1739f75853ce8 [id=1105edf1f289484c948e5568c810cdd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:25.081511 18227 leader_election.cc:290] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:25.082264 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0fbce82a1ae49409de07104f2da8c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:25.083343 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32466931871c4c5d8e9f95a11ffe26be" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:25.085188 17905 leader_election.cc:343] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:25.106160 17613 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet e0fbce82a1ae49409de07104f2da8c8f
I20260924 19:57:25.117007 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet e0fbce82a1ae49409de07104f2da8c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:25.117528 18223 ts_tablet_manager.cc:1434] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.238s user 0.005s sys 0.022s
I20260924 19:57:25.120899 18223 tablet_bootstrap.cc:492] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:25.125361 18264 raft_consensus.cc:493] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:25.131009 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet e0fbce82a1ae49409de07104f2da8c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:25.136143 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet e0fbce82a1ae49409de07104f2da8c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:25.151412 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32466931871c4c5d8e9f95a11ffe26be" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
W20260924 19:57:25.161239 17906 leader_election.cc:343] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:25.161710 17906 leader_election.cc:304] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:25.162847 18227 raft_consensus.cc:2749] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:25.151127 17802 tablet_service.cc:1511] Processing CreateTablet for tablet e0fbce82a1ae49409de07104f2da8c8f (DEFAULT_TABLE table=default.Table_810632cce6a446c5adb1739f75853ce8 [id=1105edf1f289484c948e5568c810cdd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:25.129087 18264 raft_consensus.cc:515] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.164690 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0fbce82a1ae49409de07104f2da8c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:25.166854 18224 tablet_bootstrap.cc:492] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:25.180461 18264 leader_election.cc:290] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:25.172668 18223 tablet_bootstrap.cc:654] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:25.182188 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d73260e74a324081bda4457a9e9931c9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:25.182988 17972 raft_consensus.cc:2468] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
W20260924 19:57:25.184001 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet e0fbce82a1ae49409de07104f2da8c8f because the tablet was not found. No further retry: Not found: Tablet not found: e0fbce82a1ae49409de07104f2da8c8f
I20260924 19:57:25.184291 18050 leader_election.cc:304] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:25.185081 18264 raft_consensus.cc:2804] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:25.185401 18264 raft_consensus.cc:493] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:25.185725 18264 raft_consensus.cc:3060] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:25.186359 18224 tablet_bootstrap.cc:654] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:25.158671 18231 ts_tablet_manager.cc:1929] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:25.188077 18231 log.cc:1199] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/d73260e74a324081bda4457a9e9931c9
I20260924 19:57:25.188947 18231 ts_tablet_manager.cc:1950] T d73260e74a324081bda4457a9e9931c9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
W20260924 19:57:25.192052 18051 leader_election.cc:343] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: d73260e74a324081bda4457a9e9931c9
I20260924 19:57:25.195035 18264 raft_consensus.cc:515] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.197053 18264 leader_election.cc:290] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:25.198323 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d73260e74a324081bda4457a9e9931c9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:25.198894 17972 raft_consensus.cc:3060] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:25.198910 18051 leader_election.cc:343] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: d73260e74a324081bda4457a9e9931c9
I20260924 19:57:25.201462 17613 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_88272136fce645b3aae8d87c6e82a2ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:25.203881 18220 tablet_bootstrap.cc:492] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:25.204304 18220 ts_tablet_manager.cc:1403] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.157s user 0.008s sys 0.026s
I20260924 19:57:25.204511 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet d73260e74a324081bda4457a9e9931c9 (table default.Table_d2abf2d7d4494abc8274e2ce9888b9f3 [id=e025f80a4e9744c5ba260b228f5e79ec]) successfully deleted
W20260924 19:57:25.204741 17613 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88272136fce645b3aae8d87c6e82a2ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:25.206616 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet e0fbce82a1ae49409de07104f2da8c8f because the tablet was not found. No further retry: Not found: Tablet not found: e0fbce82a1ae49409de07104f2da8c8f
I20260924 19:57:25.206760 18220 raft_consensus.cc:359] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.207384 18220 raft_consensus.cc:385] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:25.210333 17972 raft_consensus.cc:2468] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:25.211635 18050 leader_election.cc:304] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
I20260924 19:57:25.207644 18220 raft_consensus.cc:740] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:25.212831 18264 raft_consensus.cc:2804] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:25.217301 18264 raft_consensus.cc:697] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:25.217705 17612 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet ace25789ea9f47bfa723e83d5667bce7
I20260924 19:57:25.218242 18264 consensus_queue.cc:237] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.222338 18220 consensus_queue.cc:260] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.224377 18220 ts_tablet_manager.cc:1434] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260924 19:57:25.225922 18223 tablet_bootstrap.cc:492] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:25.230880 18224 tablet_bootstrap.cc:492] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:25.231976 18220 tablet_bootstrap.cc:492] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:25.236002 18223 ts_tablet_manager.cc:1403] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.009s
I20260924 19:57:25.240466 18223 raft_consensus.cc:359] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.231375 18224 ts_tablet_manager.cc:1403] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.009s
I20260924 19:57:25.251727 18223 raft_consensus.cc:385] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:25.252564 18223 raft_consensus.cc:740] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:25.254199 18223 consensus_queue.cc:260] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.261602 18223 ts_tablet_manager.cc:1434] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.024s user 0.001s sys 0.005s
I20260924 19:57:25.267493 18223 tablet_bootstrap.cc:492] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:25.274483 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet e0fbce82a1ae49409de07104f2da8c8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0fbce82a1ae49409de07104f2da8c8f already in progress: creating tablet
I20260924 19:57:25.279197 18220 tablet_bootstrap.cc:654] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:25.293031 17609 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6f7f650fc26498faff2059b42bbf91f consensus state has no leader
I20260924 19:57:25.266871 18224 raft_consensus.cc:359] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.299582 18224 raft_consensus.cc:385] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:25.299954 18224 raft_consensus.cc:740] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:25.300765 18224 consensus_queue.cc:260] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.313786 18224 ts_tablet_manager.cc:1434] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.063s user 0.007s sys 0.000s
I20260924 19:57:25.314810 18224 tablet_bootstrap.cc:492] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:25.336720 18224 tablet_bootstrap.cc:654] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:25.358253 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet d73260e74a324081bda4457a9e9931c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:25.359016 18272 tablet_replica.cc:333] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:25.359391 18224 tablet_bootstrap.cc:492] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:25.359790 18224 ts_tablet_manager.cc:1403] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.000s
I20260924 19:57:25.359767 18272 raft_consensus.cc:2243] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:25.360236 18272 raft_consensus.cc:2272] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:25.362365 18272 ts_tablet_manager.cc:1916] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:25.369022 17612 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_e251c4f5fec44bc0952684c5ff3d826f" } modify_external_catalogs: true
I20260924 19:57:25.369050 18224 raft_consensus.cc:359] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.369707 18224 raft_consensus.cc:385] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:25.369675 17612 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_e251c4f5fec44bc0952684c5ff3d826f" } modify_external_catalogs: true
I20260924 19:57:25.369966 18224 raft_consensus.cc:740] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:25.374872 18220 tablet_bootstrap.cc:492] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:25.375355 18220 ts_tablet_manager.cc:1403] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.144s user 0.019s sys 0.007s
I20260924 19:57:25.376514 18223 tablet_bootstrap.cc:654] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:25.377876 18220 raft_consensus.cc:359] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.378557 18220 raft_consensus.cc:385] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:25.378830 18220 raft_consensus.cc:740] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:25.373029 18224 consensus_queue.cc:260] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.384826 17610 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_b013bc1fcd8b4e0db6c8d928b1984a3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:25.392778 17610 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b013bc1fcd8b4e0db6c8d928b1984a3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:25.400559 18224 ts_tablet_manager.cc:1434] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20260924 19:57:25.403724 18224 tablet_bootstrap.cc:492] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:25.379513 18220 consensus_queue.cc:260] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.408849 18220 ts_tablet_manager.cc:1434] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260924 19:57:25.412837 18224 tablet_bootstrap.cc:654] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:25.425863 17615 catalog_manager.cc:5838] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:25.427372 17615 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_69b957785d9d4791aa08d3755602bb87 [id=5878a9a1b1d54055a5e8dc712b9de665]). Expected version 2 got 0
W20260924 19:57:25.428416 17608 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c6943a365da464e83cedc2b9711ca89 consensus state has no leader
I20260924 19:57:25.430750 18272 ts_tablet_manager.cc:1929] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:25.431275 18272 log.cc:1199] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/d73260e74a324081bda4457a9e9931c9
I20260924 19:57:25.432143 18272 ts_tablet_manager.cc:1950] T d73260e74a324081bda4457a9e9931c9 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:25.447295 18118 raft_consensus.cc:1275] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Refusing update from remote peer f74e6ed351284d7fb110e44336ce643a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:57:25.449323 17802 tablet_service.cc:1511] Processing CreateTablet for tablet 08aa217e7ea34aa4b839d7e992f978eb (DEFAULT_TABLE table=default.Table_87b845fa3a064c269030287021ae1c05 [id=617d557c676d462b9bf9c44c4d5310aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:25.449582 18227 consensus_queue.cc:1048] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:25.449672 17801 tablet_service.cc:1511] Processing CreateTablet for tablet f33d435d7a594344956a88506a43ef59 (DEFAULT_TABLE table=default.Table_b200a0c5e5014cc09c18338e29eb3e2a [id=fdb27eff83df4ca18ac25a6293459356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:25.450511 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08aa217e7ea34aa4b839d7e992f978eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:25.451088 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33d435d7a594344956a88506a43ef59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:25.459403 17823 raft_consensus.cc:1275] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Refusing update from remote peer f74e6ed351284d7fb110e44336ce643a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:57:25.461028 18227 consensus_queue.cc:1048] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:25.468021 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 08aa217e7ea34aa4b839d7e992f978eb (DEFAULT_TABLE table=default.Table_87b845fa3a064c269030287021ae1c05 [id=617d557c676d462b9bf9c44c4d5310aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:25.468549 18098 tablet_service.cc:1511] Processing CreateTablet for tablet f33d435d7a594344956a88506a43ef59 (DEFAULT_TABLE table=default.Table_b200a0c5e5014cc09c18338e29eb3e2a [id=fdb27eff83df4ca18ac25a6293459356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:25.469197 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08aa217e7ea34aa4b839d7e992f978eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:25.469882 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33d435d7a594344956a88506a43ef59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:25.471102 17952 tablet_service.cc:1511] Processing CreateTablet for tablet 08aa217e7ea34aa4b839d7e992f978eb (DEFAULT_TABLE table=default.Table_87b845fa3a064c269030287021ae1c05 [id=617d557c676d462b9bf9c44c4d5310aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:25.471504 17951 tablet_service.cc:1511] Processing CreateTablet for tablet f33d435d7a594344956a88506a43ef59 (DEFAULT_TABLE table=default.Table_b200a0c5e5014cc09c18338e29eb3e2a [id=fdb27eff83df4ca18ac25a6293459356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:25.472215 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08aa217e7ea34aa4b839d7e992f978eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:25.473173 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f33d435d7a594344956a88506a43ef59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:25.483711 18224 tablet_bootstrap.cc:492] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:25.487844 17614 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_87b845fa3a064c269030287021ae1c05" } modify_external_catalogs: true
I20260924 19:57:25.488309 18264 raft_consensus.cc:493] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:25.488405 17614 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_87b845fa3a064c269030287021ae1c05" } modify_external_catalogs: true
I20260924 19:57:25.488829 18264 raft_consensus.cc:515] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.491117 18264 leader_election.cc:290] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:25.491639 18224 ts_tablet_manager.cc:1403] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.088s user 0.014s sys 0.009s
I20260924 19:57:25.492678 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32466931871c4c5d8e9f95a11ffe26be" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:25.493309 17972 raft_consensus.cc:2468] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:25.494189 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet d73260e74a324081bda4457a9e9931c9 (table default.Table_d2abf2d7d4494abc8274e2ce9888b9f3 [id=e025f80a4e9744c5ba260b228f5e79ec]) successfully deleted
I20260924 19:57:25.498883 17608 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_0954c32d949b48bebae0462522b8da72" } new_table_name: "default.Table_12a91a9b08e840a3bf63db9781fcf701" modify_external_catalogs: true
I20260924 19:57:25.499562 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32466931871c4c5d8e9f95a11ffe26be" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:25.500218 17821 raft_consensus.cc:2468] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:25.501336 18051 leader_election.cc:304] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:25.501246 18224 raft_consensus.cc:359] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.501911 18224 raft_consensus.cc:385] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:25.502113 18264 raft_consensus.cc:2804] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:25.502151 18224 raft_consensus.cc:740] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:25.502445 18264 raft_consensus.cc:493] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:25.502753 18264 raft_consensus.cc:3060] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:25.503065 18224 consensus_queue.cc:260] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.504011 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_4c0023d68e2e47c8931de5a5f5b81d89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:25.505227 18224 ts_tablet_manager.cc:1434] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
W20260924 19:57:25.506544 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c0023d68e2e47c8931de5a5f5b81d89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:25.511830 18264 raft_consensus.cc:515] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.514387 18264 leader_election.cc:290] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:25.516223 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32466931871c4c5d8e9f95a11ffe26be" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:25.516939 17821 raft_consensus.cc:3060] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:25.529641 17612 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 01c515d65ada4c77a1325cc27948ec08
I20260924 19:57:25.546469 18224 tablet_bootstrap.cc:492] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:25.558396 17821 raft_consensus.cc:2468] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:25.559794 18051 leader_election.cc:304] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:25.560611 18264 raft_consensus.cc:2804] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:25.561096 18264 raft_consensus.cc:697] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:25.561906 18264 consensus_queue.cc:237] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.563141 18224 tablet_bootstrap.cc:654] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:25.563105 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_9740734f1493496d94dac20cb0e65c85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:25.565246 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9740734f1493496d94dac20cb0e65c85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:25.588002 17821 raft_consensus.cc:1275] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Refusing update from remote peer eaf64bfb64aa42948c9bc1411ce4ce35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:57:25.589581 18261 consensus_queue.cc:1048] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:25.592149 18236 raft_consensus.cc:493] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:25.592588 18236 raft_consensus.cc:515] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.598068 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6edb7999719e4c3c93b93539bfe1550c" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:25.600113 17753 leader_election.cc:343] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:25.600436 17950 tablet_service.cc:1558] Processing DeleteTablet for tablet e0fbce82a1ae49409de07104f2da8c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:25.601356 18272 tablet_replica.cc:333] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:25.602412 18272 raft_consensus.cc:2243] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:25.602995 18272 raft_consensus.cc:2272] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:25.595089 18236 leader_election.cc:290] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:25.608029 18272 ts_tablet_manager.cc:1916] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:25.610189 17971 raft_consensus.cc:1275] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Refusing update from remote peer eaf64bfb64aa42948c9bc1411ce4ce35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:57:25.611424 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32466931871c4c5d8e9f95a11ffe26be" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:25.611912 17972 raft_consensus.cc:3060] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:25.612833 18264 consensus_queue.cc:1048] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:25.616255 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6edb7999719e4c3c93b93539bfe1550c" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:25.616952 17970 raft_consensus.cc:2468] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:25.618726 17753 leader_election.cc:304] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters: eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:25.620894 18236 raft_consensus.cc:2804] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:25.621296 18236 raft_consensus.cc:493] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:25.621649 18236 raft_consensus.cc:3060] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:25.649969 17972 raft_consensus.cc:2468] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:25.656458 18223 tablet_bootstrap.cc:492] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:25.656870 18223 ts_tablet_manager.cc:1403] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.390s user 0.010s sys 0.000s
I20260924 19:57:25.658596 18220 tablet_bootstrap.cc:492] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:25.664700 18220 tablet_bootstrap.cc:654] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:25.667583 17615 catalog_manager.cc:5838] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:25.669123 17615 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 7d4bee1eaae04ce99c7684c3da321c36 (table default.Table_b06e7b3533ae4657991ac1c424cc08cd [id=965f3deebb8c4029901624f08a79febc]). Expected version 2 got 0
I20260924 19:57:25.669977 17615 catalog_manager.cc:5838] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:25.675335 18236 raft_consensus.cc:515] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.684300 17615 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet c6f7f650fc26498faff2059b42bbf91f (table default.Table_f570a48d43e84d5d86ef5a1a92e3161c [id=c3b94a6ee3b5400cb352433d44a902f4]). Expected version 2 got 0
I20260924 19:57:25.693729 18295 tablet.cc:1751] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:25.694386 17614 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 08aa217e7ea34aa4b839d7e992f978eb
I20260924 19:57:25.659096 18223 raft_consensus.cc:359] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.699927 18223 raft_consensus.cc:385] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:57:25.704063 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 92954.5us
I20260924 19:57:25.705376 17802 tablet_service.cc:1558] Processing DeleteTablet for tablet 08aa217e7ea34aa4b839d7e992f978eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:25.707805 17609 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_88272136fce645b3aae8d87c6e82a2ce" } new_table_name: "default.Table_52282978a7b94f078e3a81c982e76fbe" modify_external_catalogs: true
I20260924 19:57:25.710574 18220 tablet_bootstrap.cc:492] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:25.711233 18220 ts_tablet_manager.cc:1403] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.001s
W20260924 19:57:25.711807 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 08aa217e7ea34aa4b839d7e992f978eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08aa217e7ea34aa4b839d7e992f978eb already in progress: creating tablet
I20260924 19:57:25.713847 18220 raft_consensus.cc:359] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:25.714535 18220 raft_consensus.cc:385] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:25.714800 18220 raft_consensus.cc:740] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:25.716223 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet 08aa217e7ea34aa4b839d7e992f978eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:25.718142 18297 tablet.cc:1751] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:25.722950 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_4154e363c8fd441f8d65307c76318129"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:25.725598 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4154e363c8fd441f8d65307c76318129 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:25.727950 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6edb7999719e4c3c93b93539bfe1550c" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:25.728690 17972 raft_consensus.cc:3060] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:25.732952 17611 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b06e7b3533ae4657991ac1c424cc08cd" } modify_external_catalogs: true
I20260924 19:57:25.733603 17611 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b06e7b3533ae4657991ac1c424cc08cd" } modify_external_catalogs: true
I20260924 19:57:25.704254 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 08aa217e7ea34aa4b839d7e992f978eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:25.736599 18236 leader_election.cc:290] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:25.738989 17613 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_40cd112237824b7d8282f22c1b9afd62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:25.739337 17972 raft_consensus.cc:2468] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:25.740661 17753 leader_election.cc:304] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
W20260924 19:57:25.740815 17613 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40cd112237824b7d8282f22c1b9afd62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:25.740890 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 08aa217e7ea34aa4b839d7e992f978eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08aa217e7ea34aa4b839d7e992f978eb already in progress: creating tablet
I20260924 19:57:25.726441 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6edb7999719e4c3c93b93539bfe1550c" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:25.746588 18236 raft_consensus.cc:2804] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:25.747079 18236 raft_consensus.cc:697] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:25.748154 18223 raft_consensus.cc:740] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:25.748441 18236 consensus_queue.cc:237] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.749032 18223 consensus_queue.cc:260] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.749960 18118 raft_consensus.cc:3060] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:25.754676 17607 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_93373f379bb14b6ca7a3ac7a628cad63 [id=5878a9a1b1d54055a5e8dc712b9de665]). Expected version 3 got 0
I20260924 19:57:25.755479 18283 mvcc.cc:204] Tried to move back new op lower bound from 7332986247472717824 to 7332986245123821568. Current Snapshot: MvccSnapshot[applied={T|T < 7332986247472717824}]
I20260924 19:57:25.762614 18220 consensus_queue.cc:260] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:25.774122 18220 ts_tablet_manager.cc:1434] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.063s user 0.005s sys 0.000s
I20260924 19:57:25.790736 18292 raft_consensus.cc:493] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:25.791232 18292 raft_consensus.cc:515] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:25.793198 18292 leader_election.cc:290] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:25.794519 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08aa217e7ea34aa4b839d7e992f978eb" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:25.795166 17608 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2897faafbff460bae99990a5940d38b has no consensus state
W20260924 19:57:25.796276 17905 leader_election.cc:343] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:25.796568 17822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08aa217e7ea34aa4b839d7e992f978eb" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:25.799667 18118 raft_consensus.cc:2468] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:25.805302 18223 ts_tablet_manager.cc:1434] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.148s user 0.007s sys 0.000s
I20260924 19:57:25.806171 18223 tablet_bootstrap.cc:492] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:25.808449 18220 tablet_bootstrap.cc:492] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
W20260924 19:57:25.809049 17906 leader_election.cc:343] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:25.809470 17906 leader_election.cc:304] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:25.811111 18292 raft_consensus.cc:2749] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:25.782560 18303 tablet.cc:1751] T 0c6943a365da464e83cedc2b9711ca89 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260924 19:57:25.816160 18272 ts_tablet_manager.cc:1929] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:25.816563 18272 log.cc:1199] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/e0fbce82a1ae49409de07104f2da8c8f
I20260924 19:57:25.817240 18272 ts_tablet_manager.cc:1950] T e0fbce82a1ae49409de07104f2da8c8f P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:25.827761 17610 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet c6f7f650fc26498faff2059b42bbf91f (table default.Table_f570a48d43e84d5d86ef5a1a92e3161c [id=c3b94a6ee3b5400cb352433d44a902f4]). Expected version 2 got 0
W20260924 19:57:25.830613 17609 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8cc802bf87f4d739175f93203478af8 has no consensus state
I20260924 19:57:25.831008 18220 tablet_bootstrap.cc:654] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:25.838522 17607 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_93373f379bb14b6ca7a3ac7a628cad63 [id=5878a9a1b1d54055a5e8dc712b9de665]). Expected version 3 got 0
I20260924 19:57:25.842399 18223 tablet_bootstrap.cc:654] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:25.844453 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet e0fbce82a1ae49409de07104f2da8c8f (table default.Table_810632cce6a446c5adb1739f75853ce8 [id=1105edf1f289484c948e5568c810cdd3]) successfully deleted
W20260924 19:57:25.844707 17600 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet f2897faafbff460bae99990a5940d38b (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f2897faafbff460bae99990a5940d38b has no consensus state
I20260924 19:57:25.845881 18272 tablet_replica.cc:333] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:25.847534 18272 raft_consensus.cc:2243] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:25.855989 18272 raft_consensus.cc:2272] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:25.857926 18302 tablet.cc:1751] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260924 19:57:25.862828 18272 ts_tablet_manager.cc:1916] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:25.891064 18223 tablet_bootstrap.cc:492] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:25.891692 18223 ts_tablet_manager.cc:1403] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.012s
I20260924 19:57:25.900658 17611 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 7d4bee1eaae04ce99c7684c3da321c36
I20260924 19:57:25.902896 17607 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_f570a48d43e84d5d86ef5a1a92e3161c [id=c3b94a6ee3b5400cb352433d44a902f4] alter complete (version 2)
I20260924 19:57:25.912797 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 7d4bee1eaae04ce99c7684c3da321c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:25.915438 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet 7d4bee1eaae04ce99c7684c3da321c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:25.916199 18231 tablet_replica.cc:333] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:25.917375 18301 tablet_replica.cc:333] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:25.918148 18301 raft_consensus.cc:2243] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:25.918890 18301 pending_rounds.cc:70] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: Trying to abort 1 pending ops.
I20260924 19:57:25.919166 18301 pending_rounds.cc:77] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986245133860864 op_type: NO_OP noop_request { }
I20260924 19:57:25.919549 18301 raft_consensus.cc:2889] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:25.919885 18301 raft_consensus.cc:2272] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:25.920454 18231 raft_consensus.cc:2243] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:25.920940 18231 raft_consensus.cc:2272] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:25.922900 18231 ts_tablet_manager.cc:1916] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:25.925750 18301 ts_tablet_manager.cc:1916] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:25.926702 17951 tablet_service.cc:1558] Processing DeleteTablet for tablet 7d4bee1eaae04ce99c7684c3da321c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:25.926795 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_96afe0911af349bf9301ce80ef227bd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:25.930042 17611 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b200a0c5e5014cc09c18338e29eb3e2a" } modify_external_catalogs: true
I20260924 19:57:25.930835 17611 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b200a0c5e5014cc09c18338e29eb3e2a" } modify_external_catalogs: true
I20260924 19:57:25.934768 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet d73260e74a324081bda4457a9e9931c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:25.939340 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96afe0911af349bf9301ce80ef227bd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:25.952056 18223 raft_consensus.cc:359] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.957342 18223 raft_consensus.cc:385] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:25.959120 18223 raft_consensus.cc:740] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:25.960753 18223 consensus_queue.cc:260] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:25.961946 18220 tablet_bootstrap.cc:492] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:25.962355 18220 ts_tablet_manager.cc:1403] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.154s user 0.010s sys 0.007s
I20260924 19:57:25.964879 18220 raft_consensus.cc:359] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:25.965600 18220 raft_consensus.cc:385] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:25.965920 18220 raft_consensus.cc:740] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:25.966747 18220 consensus_queue.cc:260] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:25.972529 18281 mvcc.cc:204] Tried to move back new op lower bound from 7332986246894387200 to 7332986244697214976. Current Snapshot: MvccSnapshot[applied={T|T < 7332986246894387200}]
I20260924 19:57:25.974120 18220 ts_tablet_manager.cc:1434] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260924 19:57:25.980036 18223 ts_tablet_manager.cc:1434] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.088s user 0.003s sys 0.003s
I20260924 19:57:25.980994 18223 tablet_bootstrap.cc:492] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:25.987939 18297 tablet.cc:1746] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: Skipping requested alter to schema version 2, tablet already version 2
I20260924 19:57:25.989393 18302 tablet.cc:1746] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: Skipping requested alter to schema version 2, tablet already version 2
I20260924 19:57:25.999198 17614 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_52282978a7b94f078e3a81c982e76fbe" } new_table_name: "default.Table_3ee32d095d774dbe88f86bff9878613b" modify_external_catalogs: true
I20260924 19:57:26.013211 18231 ts_tablet_manager.cc:1929] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:26.016722 18223 tablet_bootstrap.cc:654] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.025035 17951 tablet_service.cc:1511] Processing CreateTablet for tablet 30a3acc4b104447885b27ea75dca1bea (DEFAULT_TABLE table=default.Table_b013bc1fcd8b4e0db6c8d928b1984a3b [id=0fb7eef54311401c94d0c4e97de0cc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.026381 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a3acc4b104447885b27ea75dca1bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.030074 17609 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_a86d35d7e9de4a44a991798952b6de5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:26.026293 18272 ts_tablet_manager.cc:1929] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:26.031847 18272 log.cc:1199] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/08aa217e7ea34aa4b839d7e992f978eb
W20260924 19:57:26.032413 17609 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a86d35d7e9de4a44a991798952b6de5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:26.035599 18272 ts_tablet_manager.cc:1950] T 08aa217e7ea34aa4b839d7e992f978eb P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:26.037344 17610 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_4154e363c8fd441f8d65307c76318129" } new_table_name: "default.Table_830707b707c74668ad26efcbeab482e8" modify_external_catalogs: true
I20260924 19:57:26.040901 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 30a3acc4b104447885b27ea75dca1bea (DEFAULT_TABLE table=default.Table_b013bc1fcd8b4e0db6c8d928b1984a3b [id=0fb7eef54311401c94d0c4e97de0cc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.041994 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a3acc4b104447885b27ea75dca1bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.044901 17802 tablet_service.cc:1511] Processing CreateTablet for tablet 90cd09aecb6247e7b7937996f60f007f (DEFAULT_TABLE table=default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.050096 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90cd09aecb6247e7b7937996f60f007f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.050442 18272 tablet_replica.cc:333] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:26.051266 18272 raft_consensus.cc:2243] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:26.051743 18272 raft_consensus.cc:2272] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:26.053583 18272 ts_tablet_manager.cc:1916] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:26.074481 17952 tablet_service.cc:1511] Processing CreateTablet for tablet 90cd09aecb6247e7b7937996f60f007f (DEFAULT_TABLE table=default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.075947 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90cd09aecb6247e7b7937996f60f007f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.083396 17800 tablet_service.cc:1511] Processing CreateTablet for tablet f2897faafbff460bae99990a5940d38b (DEFAULT_TABLE table=default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.084542 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2897faafbff460bae99990a5940d38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.087199 18231 log.cc:1199] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/7d4bee1eaae04ce99c7684c3da321c36
I20260924 19:57:26.088020 18231 ts_tablet_manager.cc:1950] T 7d4bee1eaae04ce99c7684c3da321c36 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:26.025007 18301 ts_tablet_manager.cc:1929] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:26.088665 18301 log.cc:1199] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/7d4bee1eaae04ce99c7684c3da321c36
I20260924 19:57:26.089471 18301 ts_tablet_manager.cc:1950] T 7d4bee1eaae04ce99c7684c3da321c36 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:26.058739 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 30a3acc4b104447885b27ea75dca1bea (DEFAULT_TABLE table=default.Table_b013bc1fcd8b4e0db6c8d928b1984a3b [id=0fb7eef54311401c94d0c4e97de0cc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.092028 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30a3acc4b104447885b27ea75dca1bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.092180 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_fca275d61c914e72b792ca0bb116d0bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:26.094349 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fca275d61c914e72b792ca0bb116d0bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:26.081254 18050 consensus_peers.cc:597] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d73260e74a324081bda4457a9e9931c9. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:26.096366 17799 tablet_service.cc:1511] Processing CreateTablet for tablet a8cc802bf87f4d739175f93203478af8 (DEFAULT_TABLE table=default.Table_52282978a7b94f078e3a81c982e76fbe [id=8d4e859f129f43b2a9a8e12861b6b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.096944 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 90cd09aecb6247e7b7937996f60f007f (DEFAULT_TABLE table=default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.097744 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8cc802bf87f4d739175f93203478af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.098359 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90cd09aecb6247e7b7937996f60f007f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.100998 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_980c453872de4a9383a8f4929e0e28f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:26.102970 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_980c453872de4a9383a8f4929e0e28f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:26.109843 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 7d4bee1eaae04ce99c7684c3da321c36 (table default.Table_b06e7b3533ae4657991ac1c424cc08cd [id=965f3deebb8c4029901624f08a79febc]) successfully deleted
I20260924 19:57:26.110652 17950 tablet_service.cc:1511] Processing CreateTablet for tablet f2897faafbff460bae99990a5940d38b (DEFAULT_TABLE table=default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.112207 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2897faafbff460bae99990a5940d38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.116966 17608 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b013bc1fcd8b4e0db6c8d928b1984a3b" } modify_external_catalogs: true
I20260924 19:57:26.117681 17608 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b013bc1fcd8b4e0db6c8d928b1984a3b" } modify_external_catalogs: true
I20260924 19:57:26.098333 18301 tablet_replica.cc:333] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:26.123955 18301 raft_consensus.cc:2243] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:26.125124 18301 pending_rounds.cc:70] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: Trying to abort 1 pending ops.
I20260924 19:57:26.125437 18301 pending_rounds.cc:77] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986246022934528 op_type: NO_OP noop_request { }
I20260924 19:57:26.125878 18301 raft_consensus.cc:2889] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:26.126278 18301 raft_consensus.cc:2272] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:26.128994 18301 ts_tablet_manager.cc:1916] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:26.152925 18223 tablet_bootstrap.cc:492] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:26.153460 18223 ts_tablet_manager.cc:1403] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.173s user 0.010s sys 0.001s
I20260924 19:57:26.156090 18223 raft_consensus.cc:359] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:26.157778 17614 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8cc802bf87f4d739175f93203478af8 consensus state has no leader
I20260924 19:57:26.163270 18223 raft_consensus.cc:385] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.163563 18223 raft_consensus.cc:740] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.167500 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 08aa217e7ea34aa4b839d7e992f978eb (table default.Table_87b845fa3a064c269030287021ae1c05 [id=617d557c676d462b9bf9c44c4d5310aa]) successfully deleted
I20260924 19:57:26.170145 18097 tablet_service.cc:1511] Processing CreateTablet for tablet f2897faafbff460bae99990a5940d38b (DEFAULT_TABLE table=default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.171602 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2897faafbff460bae99990a5940d38b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.171422 18223 consensus_queue.cc:260] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:26.177130 17949 tablet_service.cc:1511] Processing CreateTablet for tablet a8cc802bf87f4d739175f93203478af8 (DEFAULT_TABLE table=default.Table_52282978a7b94f078e3a81c982e76fbe [id=8d4e859f129f43b2a9a8e12861b6b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.178342 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8cc802bf87f4d739175f93203478af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.185453 18096 tablet_service.cc:1511] Processing CreateTablet for tablet a8cc802bf87f4d739175f93203478af8 (DEFAULT_TABLE table=default.Table_52282978a7b94f078e3a81c982e76fbe [id=8d4e859f129f43b2a9a8e12861b6b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.186545 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8cc802bf87f4d739175f93203478af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.191548 18223 ts_tablet_manager.cc:1434] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20260924 19:57:26.199518 18223 tablet_bootstrap.cc:492] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:26.204245 18300 tablet.cc:1751] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260924 19:57:26.211792 17823 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.003s sys 0.000s
W20260924 19:57:26.212169 17823 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s user 0.003s sys 0.000s
I20260924 19:57:26.215299 18272 ts_tablet_manager.cc:1929] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:26.216481 18272 log.cc:1199] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/7d4bee1eaae04ce99c7684c3da321c36
I20260924 19:57:26.218329 18272 ts_tablet_manager.cc:1950] T 7d4bee1eaae04ce99c7684c3da321c36 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:26.220377 18223 tablet_bootstrap.cc:654] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:26.220863 18224 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.007s sys 0.000s
W20260924 19:57:26.221186 18224 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.643s user 0.007s sys 0.000s
I20260924 19:57:26.223035 18224 tablet_bootstrap.cc:492] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:26.223518 18224 ts_tablet_manager.cc:1403] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.677s user 0.018s sys 0.001s
I20260924 19:57:26.234174 18301 ts_tablet_manager.cc:1929] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:26.234746 18301 log.cc:1199] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/d73260e74a324081bda4457a9e9931c9
I20260924 19:57:26.235627 18301 ts_tablet_manager.cc:1950] T d73260e74a324081bda4457a9e9931c9 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:26.231999 18224 raft_consensus.cc:359] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:26.242712 18224 raft_consensus.cc:385] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.243059 18224 raft_consensus.cc:740] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.234151 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 7d4bee1eaae04ce99c7684c3da321c36 (table default.Table_b06e7b3533ae4657991ac1c424cc08cd [id=965f3deebb8c4029901624f08a79febc]) successfully deleted
I20260924 19:57:26.245076 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet d73260e74a324081bda4457a9e9931c9 (table default.Table_d2abf2d7d4494abc8274e2ce9888b9f3 [id=e025f80a4e9744c5ba260b228f5e79ec]) successfully deleted
I20260924 19:57:26.245801 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 7d4bee1eaae04ce99c7684c3da321c36 (table default.Table_b06e7b3533ae4657991ac1c424cc08cd [id=965f3deebb8c4029901624f08a79febc]) successfully deleted
I20260924 19:57:26.251968 17611 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet f33d435d7a594344956a88506a43ef59
W20260924 19:57:26.254478 17610 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83ee43b6dc064dc7b320691d52e57ea7 has no consensus state
I20260924 19:57:26.257692 18224 consensus_queue.cc:260] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:26.259800 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet f33d435d7a594344956a88506a43ef59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:26.260268 17615 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_93373f379bb14b6ca7a3ac7a628cad63 [id=5878a9a1b1d54055a5e8dc712b9de665] alter complete (version 3)
I20260924 19:57:26.260985 18223 tablet_bootstrap.cc:492] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:26.261389 18223 ts_tablet_manager.cc:1403] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.002s
I20260924 19:57:26.261587 18224 ts_tablet_manager.cc:1434] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20260924 19:57:26.262382 18224 tablet_bootstrap.cc:492] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:26.265244 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet f33d435d7a594344956a88506a43ef59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f33d435d7a594344956a88506a43ef59 already in progress: creating tablet
I20260924 19:57:26.266916 18223 raft_consensus.cc:359] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:26.268790 18223 raft_consensus.cc:385] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.269079 18223 raft_consensus.cc:740] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.269717 18223 consensus_queue.cc:260] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:26.283605 18220 tablet_bootstrap.cc:492] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:26.286418 18223 ts_tablet_manager.cc:1434] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260924 19:57:26.290134 18220 tablet_bootstrap.cc:654] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.301242 18223 tablet_bootstrap.cc:492] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:26.307284 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet f33d435d7a594344956a88506a43ef59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:26.308199 18272 tablet_replica.cc:333] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:26.309010 18272 raft_consensus.cc:2243] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:26.309490 18272 raft_consensus.cc:2272] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:26.309737 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet 08aa217e7ea34aa4b839d7e992f978eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:26.310266 18096 tablet_service.cc:1558] Processing DeleteTablet for tablet e0fbce82a1ae49409de07104f2da8c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:26.310509 18301 tablet_replica.cc:333] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:26.311326 18301 raft_consensus.cc:2243] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:26.311398 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet f33d435d7a594344956a88506a43ef59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:26.311807 18301 raft_consensus.cc:2272] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:26.312220 18224 tablet_bootstrap.cc:654] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.313287 18272 ts_tablet_manager.cc:1916] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:26.314218 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_e3a6abe9c34a4fd6a8567a9c0a8190c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:26.316033 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3a6abe9c34a4fd6a8567a9c0a8190c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:26.321640 17608 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 30a3acc4b104447885b27ea75dca1bea
I20260924 19:57:26.325477 18300 tablet.cc:1746] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: Skipping requested alter to schema version 2, tablet already version 2
I20260924 19:57:26.328255 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet 30a3acc4b104447885b27ea75dca1bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:26.330107 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 30a3acc4b104447885b27ea75dca1bea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30a3acc4b104447885b27ea75dca1bea already in progress: creating tablet
I20260924 19:57:26.335012 18272 ts_tablet_manager.cc:1929] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:26.335503 18272 log.cc:1199] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/f33d435d7a594344956a88506a43ef59
I20260924 19:57:26.336290 18272 ts_tablet_manager.cc:1950] T f33d435d7a594344956a88506a43ef59 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:26.340737 18220 tablet_bootstrap.cc:492] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:26.341163 18220 ts_tablet_manager.cc:1403] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.014s
I20260924 19:57:26.342584 18223 tablet_bootstrap.cc:654] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.343559 18220 raft_consensus.cc:359] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.344139 18220 raft_consensus.cc:385] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.344430 18220 raft_consensus.cc:740] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.345232 18220 consensus_queue.cc:260] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.346797 18301 ts_tablet_manager.cc:1916] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:26.356622 18220 ts_tablet_manager.cc:1434] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
W20260924 19:57:26.359686 17602 consensus_peers.cc:597] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e -> Peer 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309): Couldn't send request to peer 3e059bbe4dac4178bccb3be48cb1118d. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260924 19:57:26.360881 18220 tablet_bootstrap.cc:492] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:26.363864 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.190:46309
--webserver_interface=127.17.2.190
--webserver_port=37347
--builtin_ntp_servers=127.17.2.148:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260924 19:57:26.364884 17612 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_830707b707c74668ad26efcbeab482e8" } new_table_name: "default.Table_b701796a010f4e5a91d2314918867a9b" modify_external_catalogs: true
I20260924 19:57:26.367121 18220 tablet_bootstrap.cc:654] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.371320 18288 raft_consensus.cc:493] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:26.371865 18288 raft_consensus.cc:515] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:26.372807 17949 tablet_service.cc:1558] Processing DeleteTablet for tablet 30a3acc4b104447885b27ea75dca1bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:26.374070 18288 leader_election.cc:290] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:26.374802 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 30a3acc4b104447885b27ea75dca1bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:26.377393 18050 leader_election.cc:343] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Not found: Tablet not found: f33d435d7a594344956a88506a43ef59
I20260924 19:57:26.379290 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f33d435d7a594344956a88506a43ef59" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:26.381052 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet f33d435d7a594344956a88506a43ef59 (table default.Table_b200a0c5e5014cc09c18338e29eb3e2a [id=fdb27eff83df4ca18ac25a6293459356]) successfully deleted
W20260924 19:57:26.381564 18051 leader_election.cc:343] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:26.381954 18051 leader_election.cc:304] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:26.386776 17609 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_93373f379bb14b6ca7a3ac7a628cad63 [id=5878a9a1b1d54055a5e8dc712b9de665]). Expected version 3 got 0
W20260924 19:57:26.382208 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 30a3acc4b104447885b27ea75dca1bea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30a3acc4b104447885b27ea75dca1bea already in progress: creating tablet
I20260924 19:57:26.393199 17613 master_service.cc:960] ReplaceTablet: received request to replace tablet 08aa217e7ea34aa4b839d7e992f978eb from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:26.397498 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet e0fbce82a1ae49409de07104f2da8c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:24 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:26.398298 18231 tablet_replica.cc:333] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:26.399101 18231 raft_consensus.cc:2243] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:26.399561 18231 raft_consensus.cc:2272] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:26.401279 18231 ts_tablet_manager.cc:1916] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:26.411420 17609 master_service.cc:960] ReplaceTablet: received request to replace tablet 90cd09aecb6247e7b7937996f60f007f from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:26.416847 18224 tablet_bootstrap.cc:492] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:26.417269 18224 ts_tablet_manager.cc:1403] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.155s user 0.007s sys 0.004s
I20260924 19:57:26.421990 18224 raft_consensus.cc:359] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:26.422657 18224 raft_consensus.cc:385] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.422940 18224 raft_consensus.cc:740] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.423820 18224 consensus_queue.cc:260] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:26.426702 18224 ts_tablet_manager.cc:1434] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260924 19:57:26.428304 18224 tablet_bootstrap.cc:492] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:26.435555 18224 tablet_bootstrap.cc:654] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.438422 18321 raft_consensus.cc:2749] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:26.439877 18231 ts_tablet_manager.cc:1929] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:26.440235 18231 log.cc:1199] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/e0fbce82a1ae49409de07104f2da8c8f
I20260924 19:57:26.440920 18231 ts_tablet_manager.cc:1950] T e0fbce82a1ae49409de07104f2da8c8f P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
W20260924 19:57:26.452328 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 69874.8us
I20260924 19:57:26.456532 18300 tablet.cc:1751] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260924 19:57:26.461696 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet e0fbce82a1ae49409de07104f2da8c8f (table default.Table_810632cce6a446c5adb1739f75853ce8 [id=1105edf1f289484c948e5568c810cdd3]) successfully deleted
I20260924 19:57:26.481346 17611 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_980c453872de4a9383a8f4929e0e28f3" } new_table_name: "default.Table_1a379ab4fc834d69865de7106e463200" modify_external_catalogs: true
I20260924 19:57:26.483932 18325 raft_consensus.cc:493] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:26.484485 18325 raft_consensus.cc:515] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:26.486183 17610 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_f570a48d43e84d5d86ef5a1a92e3161c" } modify_external_catalogs: true
I20260924 19:57:26.488072 17610 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_f570a48d43e84d5d86ef5a1a92e3161c" } modify_external_catalogs: true
W20260924 19:57:26.489971 18050 leader_election.cc:343] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Not found: Tablet not found: e0fbce82a1ae49409de07104f2da8c8f
I20260924 19:57:26.492769 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_850acd445f004facb3a20aaac28003e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:26.494500 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_850acd445f004facb3a20aaac28003e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:26.499264 18325 leader_election.cc:290] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:26.501345 18224 tablet_bootstrap.cc:492] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:26.501747 18224 ts_tablet_manager.cc:1403] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.000s
I20260924 19:57:26.505942 18223 tablet_bootstrap.cc:492] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:26.506383 18223 ts_tablet_manager.cc:1403] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.205s user 0.009s sys 0.000s
I20260924 19:57:26.507920 18321 consensus_queue.cc:1048] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:26.518172 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_b814c73599bb4d9fae83012912129997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:26.520030 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b814c73599bb4d9fae83012912129997 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:26.524065 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 48644.7us
W20260924 19:57:26.526885 18051 leader_election.cc:343] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: e0fbce82a1ae49409de07104f2da8c8f
I20260924 19:57:26.528246 18051 leader_election.cc:304] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:26.529022 18321 raft_consensus.cc:2749] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:26.532354 18223 raft_consensus.cc:359] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.532958 18223 raft_consensus.cc:385] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.533200 18223 raft_consensus.cc:740] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.537631 18322 consensus_queue.cc:1048] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:26.517794 18224 raft_consensus.cc:359] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.538851 18224 raft_consensus.cc:385] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.556857 18224 raft_consensus.cc:740] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.540233 18223 consensus_queue.cc:260] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.564277 18224 consensus_queue.cc:260] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.571885 18224 ts_tablet_manager.cc:1434] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.070s user 0.007s sys 0.001s
W20260924 19:57:26.573195 17612 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83ee43b6dc064dc7b320691d52e57ea7 has no consensus state
W20260924 19:57:26.574376 17611 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a384c9a0bb65429ebb97fe0807c78773 has no consensus state
I20260924 19:57:26.577127 18223 ts_tablet_manager.cc:1434] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.066s user 0.004s sys 0.001s
I20260924 19:57:26.583145 18223 tablet_bootstrap.cc:492] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:26.587555 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet f2897faafbff460bae99990a5940d38b (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:26.587736 18301 ts_tablet_manager.cc:1929] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:26.588120 18301 log.cc:1199] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/08aa217e7ea34aa4b839d7e992f978eb
I20260924 19:57:26.588798 18301 ts_tablet_manager.cc:1950] T 08aa217e7ea34aa4b839d7e992f978eb P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:26.596998 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 08aa217e7ea34aa4b839d7e992f978eb (table default.Table_87b845fa3a064c269030287021ae1c05 [id=617d557c676d462b9bf9c44c4d5310aa]) successfully deleted
I20260924 19:57:26.597201 18223 tablet_bootstrap.cc:654] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.598702 17949 tablet_service.cc:1511] Processing CreateTablet for tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 (DEFAULT_TABLE table=default.Table_96afe0911af349bf9301ce80ef227bd0 [id=8edea4510fd84d40819b48b401ccfe64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.599507 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 (DEFAULT_TABLE table=default.Table_96afe0911af349bf9301ce80ef227bd0 [id=8edea4510fd84d40819b48b401ccfe64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.599828 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.600541 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.600623 18224 tablet_bootstrap.cc:492] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:26.603148 17799 tablet_service.cc:1511] Processing CreateTablet for tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 (DEFAULT_TABLE table=default.Table_96afe0911af349bf9301ce80ef227bd0 [id=8edea4510fd84d40819b48b401ccfe64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.603937 17951 tablet_service.cc:1511] Processing CreateTablet for tablet f1d456a7ccbe4f03b41d6b52ae382424 (DEFAULT_TABLE table=default.Table_40cd112237824b7d8282f22c1b9afd62 [id=ef215e1163d74af69a3a3f6f763c3249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.604240 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.604971 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1d456a7ccbe4f03b41d6b52ae382424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.611454 18290 raft_consensus.cc:493] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:26.611907 18290 raft_consensus.cc:515] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:26.613183 17611 master_service.cc:960] ReplaceTablet: received request to replace tablet 08aa217e7ea34aa4b839d7e992f978eb from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:26.613479 18220 tablet_bootstrap.cc:492] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:26.614032 18220 ts_tablet_manager.cc:1403] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.253s user 0.026s sys 0.021s
I20260924 19:57:26.614032 18290 leader_election.cc:290] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:26.615316 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f33d435d7a594344956a88506a43ef59" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:26.615913 18118 raft_consensus.cc:2468] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:26.617856 17753 leader_election.cc:304] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
W20260924 19:57:26.618737 17753 leader_election.cc:343] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Not found: Tablet not found: f33d435d7a594344956a88506a43ef59
I20260924 19:57:26.619494 18290 raft_consensus.cc:2804] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:26.619827 18290 raft_consensus.cc:493] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:26.620134 18290 raft_consensus.cc:3060] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:26.620752 17801 tablet_service.cc:1511] Processing CreateTablet for tablet f1d456a7ccbe4f03b41d6b52ae382424 (DEFAULT_TABLE table=default.Table_40cd112237824b7d8282f22c1b9afd62 [id=ef215e1163d74af69a3a3f6f763c3249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.621824 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1d456a7ccbe4f03b41d6b52ae382424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.622682 18096 tablet_service.cc:1511] Processing CreateTablet for tablet f1d456a7ccbe4f03b41d6b52ae382424 (DEFAULT_TABLE table=default.Table_40cd112237824b7d8282f22c1b9afd62 [id=ef215e1163d74af69a3a3f6f763c3249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.623771 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1d456a7ccbe4f03b41d6b52ae382424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.627512 17952 tablet_service.cc:1511] Processing CreateTablet for tablet 83ee43b6dc064dc7b320691d52e57ea7 (DEFAULT_TABLE table=default.Table_b701796a010f4e5a91d2314918867a9b [id=31e1c3788d264823a12ecd9755ec2027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.628705 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83ee43b6dc064dc7b320691d52e57ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.623778 18220 raft_consensus.cc:359] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.631651 18220 raft_consensus.cc:385] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.631918 18220 raft_consensus.cc:740] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.632664 18224 tablet_bootstrap.cc:654] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.634018 18301 tablet_replica.cc:333] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:26.635105 18301 raft_consensus.cc:2243] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:26.635607 18301 raft_consensus.cc:2272] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:26.637507 18301 ts_tablet_manager.cc:1916] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:26.638230 17802 tablet_service.cc:1511] Processing CreateTablet for tablet 83ee43b6dc064dc7b320691d52e57ea7 (DEFAULT_TABLE table=default.Table_b701796a010f4e5a91d2314918867a9b [id=31e1c3788d264823a12ecd9755ec2027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.639426 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83ee43b6dc064dc7b320691d52e57ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.641614 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 83ee43b6dc064dc7b320691d52e57ea7 (DEFAULT_TABLE table=default.Table_b701796a010f4e5a91d2314918867a9b [id=31e1c3788d264823a12ecd9755ec2027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.642710 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83ee43b6dc064dc7b320691d52e57ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.648774 18236 consensus_queue.cc:1048] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:26.659394 18290 raft_consensus.cc:515] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:26.667410 17753 leader_election.cc:343] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Not found: Tablet not found: f33d435d7a594344956a88506a43ef59
I20260924 19:57:26.669099 18290 leader_election.cc:290] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:26.669569 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f33d435d7a594344956a88506a43ef59" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:26.670146 18117 raft_consensus.cc:3060] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:26.670002 17950 tablet_service.cc:1511] Processing CreateTablet for tablet a9505fd426134d068b95639012112bf3 (DEFAULT_TABLE table=default.Table_9740734f1493496d94dac20cb0e65c85 [id=0ceede10f5ea46729576babbd0df0c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.671132 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9505fd426134d068b95639012112bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.632571 18220 consensus_queue.cc:260] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.675025 18220 ts_tablet_manager.cc:1434] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.061s user 0.005s sys 0.000s
I20260924 19:57:26.675908 18220 tablet_bootstrap.cc:492] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:26.677330 18117 raft_consensus.cc:2468] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:26.678580 17753 leader_election.cc:304] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:26.688649 17800 tablet_service.cc:1511] Processing CreateTablet for tablet a9505fd426134d068b95639012112bf3 (DEFAULT_TABLE table=default.Table_9740734f1493496d94dac20cb0e65c85 [id=0ceede10f5ea46729576babbd0df0c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.689838 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9505fd426134d068b95639012112bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.694881 18236 raft_consensus.cc:2804] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:26.697383 18223 tablet_bootstrap.cc:492] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:26.697904 18223 ts_tablet_manager.cc:1403] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.115s user 0.010s sys 0.002s
I20260924 19:57:26.700338 17952 tablet_service.cc:1511] Processing CreateTablet for tablet fb011ae3f182464ab68d77924ab70181 (DEFAULT_TABLE table=default.Table_4c0023d68e2e47c8931de5a5f5b81d89 [id=2d056489f9314bb59c83e506d47ca5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.701351 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb011ae3f182464ab68d77924ab70181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.704421 17798 tablet_service.cc:1511] Processing CreateTablet for tablet fb011ae3f182464ab68d77924ab70181 (DEFAULT_TABLE table=default.Table_4c0023d68e2e47c8931de5a5f5b81d89 [id=2d056489f9314bb59c83e506d47ca5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.705698 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb011ae3f182464ab68d77924ab70181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.707122 18095 tablet_service.cc:1511] Processing CreateTablet for tablet fb011ae3f182464ab68d77924ab70181 (DEFAULT_TABLE table=default.Table_4c0023d68e2e47c8931de5a5f5b81d89 [id=2d056489f9314bb59c83e506d47ca5b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.708374 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb011ae3f182464ab68d77924ab70181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.709247 18224 tablet_bootstrap.cc:492] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:26.709661 18224 ts_tablet_manager.cc:1403] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.004s
I20260924 19:57:26.714599 18220 tablet_bootstrap.cc:654] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.714159 18224 raft_consensus.cc:359] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.722461 18224 raft_consensus.cc:385] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.722735 18224 raft_consensus.cc:740] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.700670 18223 raft_consensus.cc:359] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.729596 18223 raft_consensus.cc:385] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.729921 18223 raft_consensus.cc:740] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.730103 18224 consensus_queue.cc:260] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.731051 18097 tablet_service.cc:1511] Processing CreateTablet for tablet a9505fd426134d068b95639012112bf3 (DEFAULT_TABLE table=default.Table_9740734f1493496d94dac20cb0e65c85 [id=0ceede10f5ea46729576babbd0df0c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:26.732522 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9505fd426134d068b95639012112bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:26.743024 17606 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_88bf80a74c3245d1a74b3757ef4eae71" } modify_external_catalogs: true
I20260924 19:57:26.743647 17606 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_88bf80a74c3245d1a74b3757ef4eae71" } modify_external_catalogs: true
I20260924 19:57:26.746240 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_9a7c568838e54aa8b0ea15eeb70d7cd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:26.747982 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a7c568838e54aa8b0ea15eeb70d7cd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:26.755035 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet f2897faafbff460bae99990a5940d38b (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:26.757979 18236 raft_consensus.cc:697] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:26.769189 18290 consensus_queue.cc:1048] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:26.753720 18223 consensus_queue.cc:260] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.780239 18322 raft_consensus.cc:493] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:26.780797 17614 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_40cd112237824b7d8282f22c1b9afd62" } new_table_name: "default.Table_4a15e8e61f934d08b802fd02d421cd3f" modify_external_catalogs: true
I20260924 19:57:26.780694 18322 raft_consensus.cc:515] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.785168 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90cd09aecb6247e7b7937996f60f007f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:26.785446 18322 leader_election.cc:290] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:26.785790 17972 raft_consensus.cc:2468] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:26.786584 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90cd09aecb6247e7b7937996f60f007f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:26.787209 17823 raft_consensus.cc:2468] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:26.787503 18050 leader_election.cc:304] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:26.758790 18236 consensus_queue.cc:237] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:26.792030 18322 raft_consensus.cc:2804] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:26.792490 18322 raft_consensus.cc:493] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:26.792758 18223 ts_tablet_manager.cc:1434] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.094s user 0.006s sys 0.000s
I20260924 19:57:26.792887 18322 raft_consensus.cc:3060] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:26.799399 18224 ts_tablet_manager.cc:1434] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.089s user 0.013s sys 0.003s
I20260924 19:57:26.800313 18224 tablet_bootstrap.cc:492] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:26.799520 18223 tablet_bootstrap.cc:492] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:26.812654 18223 tablet_bootstrap.cc:654] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.819916 18220 tablet_bootstrap.cc:492] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:26.803597 18322 raft_consensus.cc:515] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.827378 18220 ts_tablet_manager.cc:1403] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.152s user 0.012s sys 0.000s
I20260924 19:57:26.831519 18220 raft_consensus.cc:359] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.834893 18223 tablet_bootstrap.cc:492] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:26.835206 18301 ts_tablet_manager.cc:1929] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:26.835807 18322 leader_election.cc:290] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:26.836938 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90cd09aecb6247e7b7937996f60f007f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:26.837484 17823 raft_consensus.cc:3060] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:26.839296 18224 tablet_bootstrap.cc:654] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.845232 17823 raft_consensus.cc:2468] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:26.846410 18051 leader_election.cc:304] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:26.835317 18223 ts_tablet_manager.cc:1403] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.000s
I20260924 19:57:26.849476 18223 raft_consensus.cc:359] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.855275 18223 raft_consensus.cc:385] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.855610 18223 raft_consensus.cc:740] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.835626 18301 log.cc:1199] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/e0fbce82a1ae49409de07104f2da8c8f
I20260924 19:57:26.856777 18301 ts_tablet_manager.cc:1950] T e0fbce82a1ae49409de07104f2da8c8f P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:26.859408 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet e0fbce82a1ae49409de07104f2da8c8f (table default.Table_810632cce6a446c5adb1739f75853ce8 [id=1105edf1f289484c948e5568c810cdd3]) successfully deleted
I20260924 19:57:26.860630 18301 tablet_replica.cc:333] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:26.862438 18322 raft_consensus.cc:2804] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:26.863406 18301 raft_consensus.cc:2243] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 19:57:26.863374 17602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.190:46309, user_credentials={real_user=slave}} blocked reactor thread for 53691.9us
I20260924 19:57:26.865415 18322 raft_consensus.cc:697] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:26.872963 18322 consensus_queue.cc:237] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.861100 18223 consensus_queue.cc:260] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.873987 18301 raft_consensus.cc:2272] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:26.879654 18301 ts_tablet_manager.cc:1916] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:26.834072 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90cd09aecb6247e7b7937996f60f007f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:26.881562 17972 raft_consensus.cc:3060] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:26.887887 17972 raft_consensus.cc:2468] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:26.887908 17607 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_4c0023d68e2e47c8931de5a5f5b81d89" } new_table_name: "default.Table_3d0e38cde76d4300a129a04a04b5915d" modify_external_catalogs: true
I20260924 19:57:26.889631 18220 raft_consensus.cc:385] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.880508 18223 ts_tablet_manager.cc:1434] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260924 19:57:26.889915 18220 raft_consensus.cc:740] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.891835 17613 catalog_manager.cc:5838] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:26.893301 18223 tablet_bootstrap.cc:492] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:26.894800 18236 raft_consensus.cc:493] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:26.895263 18236 raft_consensus.cc:515] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:26.897317 18236 leader_election.cc:290] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:26.902508 17610 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet c6f7f650fc26498faff2059b42bbf91f
I20260924 19:57:26.904927 17610 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_388c160758d44e1e89b8cb5d541316f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:26.907663 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet c6f7f650fc26498faff2059b42bbf91f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:26.908432 18231 tablet_replica.cc:333] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:26.909210 18231 raft_consensus.cc:2243] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:26.909720 18231 raft_consensus.cc:2272] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:26.910995 17753 leader_election.cc:343] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 08aa217e7ea34aa4b839d7e992f978eb
I20260924 19:57:26.912818 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet c6f7f650fc26498faff2059b42bbf91f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:26.913548 17950 tablet_service.cc:1558] Processing DeleteTablet for tablet c6f7f650fc26498faff2059b42bbf91f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:26.914508 17753 leader_election.cc:343] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Not found: Tablet not found: 08aa217e7ea34aa4b839d7e992f978eb
I20260924 19:57:26.914909 17753 leader_election.cc:304] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:26.915237 18231 ts_tablet_manager.cc:1916] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:26.916316 18236 raft_consensus.cc:2749] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:26.913626 17610 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_388c160758d44e1e89b8cb5d541316f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:26.917550 17599 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet a384c9a0bb65429ebb97fe0807c78773 (table default.Table_1a379ab4fc834d69865de7106e463200 [id=3ee2ed8b290747128a29f7c107dbfe82], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a384c9a0bb65429ebb97fe0807c78773 has no consensus state
I20260924 19:57:26.917873 18223 tablet_bootstrap.cc:654] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.936940 18224 tablet_bootstrap.cc:492] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:26.937390 18224 ts_tablet_manager.cc:1403] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.137s user 0.001s sys 0.008s
I20260924 19:57:26.941887 18224 raft_consensus.cc:359] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.942498 18224 raft_consensus.cc:385] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.942752 18224 raft_consensus.cc:740] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:26.943842 18224 consensus_queue.cc:260] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.950975 18224 ts_tablet_manager.cc:1434] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260924 19:57:26.952157 18224 tablet_bootstrap.cc:492] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:26.957286 18224 tablet_bootstrap.cc:654] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:26.897060 18220 consensus_queue.cc:260] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.960963 18220 ts_tablet_manager.cc:1434] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.131s user 0.005s sys 0.000s
I20260924 19:57:26.961886 18220 tablet_bootstrap.cc:492] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:26.968900 18223 tablet_bootstrap.cc:492] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:26.969586 18223 ts_tablet_manager.cc:1403] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.000s
I20260924 19:57:26.972681 18223 raft_consensus.cc:359] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.973315 18223 raft_consensus.cc:385] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:26.973562 18223 raft_consensus.cc:740] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
W20260924 19:57:26.976387 17906 consensus_peers.cc:597] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:26.980953 17609 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 90cd09aecb6247e7b7937996f60f007f
I20260924 19:57:26.981345 18223 consensus_queue.cc:260] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.982525 17609 catalog_manager.cc:6934] ReplaceTablet: tablet 90cd09aecb6247e7b7937996f60f007f deleted and replaced by tablet efd7339d0fee436299a7eac02a82addd
I20260924 19:57:26.983511 18236 raft_consensus.cc:493] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:26.983836 17614 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1d456a7ccbe4f03b41d6b52ae382424 consensus state has no leader
I20260924 19:57:26.983947 18236 raft_consensus.cc:515] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:26.985777 18236 leader_election.cc:290] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:26.988040 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30a3acc4b104447885b27ea75dca1bea" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:26.988770 18118 raft_consensus.cc:2468] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:26.989454 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 90cd09aecb6247e7b7937996f60f007f with delete_type TABLET_DATA_DELETED (replaced by tablet efd7339d0fee436299a7eac02a82addd) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:26.990187 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30a3acc4b104447885b27ea75dca1bea" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:26.990809 17972 raft_consensus.cc:2468] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:26.991369 18223 ts_tablet_manager.cc:1434] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260924 19:57:26.992183 18223 tablet_bootstrap.cc:492] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:26.996853 17753 leader_election.cc:304] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:26.997244 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 90cd09aecb6247e7b7937996f60f007f with delete_type TABLET_DATA_DELETED (replaced by tablet efd7339d0fee436299a7eac02a82addd) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:26.998176 18337 raft_consensus.cc:493] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:26.998248 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet 90cd09aecb6247e7b7937996f60f007f with delete_type TABLET_DATA_DELETED (replaced by tablet efd7339d0fee436299a7eac02a82addd) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:26.998570 18337 raft_consensus.cc:515] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:27.000653 18337 leader_election.cc:290] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:27.001143 18337 raft_consensus.cc:2804] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:27.001441 18337 raft_consensus.cc:493] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:27.001744 18337 raft_consensus.cc:3060] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.000105 18223 tablet_bootstrap.cc:654] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.003540 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_e10b7d83fa124717bf2c0f53a065500b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:27.007647 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2897faafbff460bae99990a5940d38b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:27.008275 17972 raft_consensus.cc:2468] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
W20260924 19:57:27.008348 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e10b7d83fa124717bf2c0f53a065500b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:27.009351 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2897faafbff460bae99990a5940d38b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:27.009924 18118 raft_consensus.cc:2468] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:27.012079 17753 leader_election.cc:304] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:27.018018 18236 raft_consensus.cc:2804] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:27.018380 18236 raft_consensus.cc:493] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:27.018684 18236 raft_consensus.cc:3060] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.020140 18301 ts_tablet_manager.cc:1929] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:27.020560 18301 log.cc:1199] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/f33d435d7a594344956a88506a43ef59
I20260924 19:57:27.021248 18301 ts_tablet_manager.cc:1950] T f33d435d7a594344956a88506a43ef59 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
W20260924 19:57:27.034964 17607 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb011ae3f182464ab68d77924ab70181 consensus state has no leader
I20260924 19:57:27.043767 18236 raft_consensus.cc:515] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:27.046427 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2897faafbff460bae99990a5940d38b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:27.046964 17972 raft_consensus.cc:3060] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.051972 18301 tablet_replica.cc:333] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:27.052160 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet f33d435d7a594344956a88506a43ef59 (table default.Table_b200a0c5e5014cc09c18338e29eb3e2a [id=fdb27eff83df4ca18ac25a6293459356]) successfully deleted
I20260924 19:57:27.052798 18301 raft_consensus.cc:2243] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:27.053422 18301 raft_consensus.cc:2272] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.054265 17972 raft_consensus.cc:2468] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:27.055491 17753 leader_election.cc:304] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:27.056300 18290 raft_consensus.cc:2804] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:27.056725 18290 raft_consensus.cc:697] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:27.057472 18290 consensus_queue.cc:237] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:27.058387 18220 tablet_bootstrap.cc:654] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.063503 18301 ts_tablet_manager.cc:1916] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.042425 18337 raft_consensus.cc:515] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:27.073472 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30a3acc4b104447885b27ea75dca1bea" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:27.074116 17970 raft_consensus.cc:3060] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.075771 18231 ts_tablet_manager.cc:1929] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260924 19:57:27.076134 18231 log.cc:1199] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/c6f7f650fc26498faff2059b42bbf91f
I20260924 19:57:27.076857 18231 ts_tablet_manager.cc:1950] T c6f7f650fc26498faff2059b42bbf91f P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:27.079658 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet c6f7f650fc26498faff2059b42bbf91f (table default.Table_f570a48d43e84d5d86ef5a1a92e3161c [id=c3b94a6ee3b5400cb352433d44a902f4]) successfully deleted
I20260924 19:57:27.082523 17970 raft_consensus.cc:2468] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:27.084960 18231 tablet_replica.cc:333] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:27.084805 17753 leader_election.cc:304] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:27.085963 18236 leader_election.cc:290] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:27.086437 18290 raft_consensus.cc:2804] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:27.086833 18337 leader_election.cc:290] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:27.086966 18290 raft_consensus.cc:697] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:27.087924 18290 consensus_queue.cc:237] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:27.092310 18348 tablet_replica.cc:333] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:27.092133 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2897faafbff460bae99990a5940d38b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:27.092741 18118 raft_consensus.cc:3060] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.093330 18348 raft_consensus.cc:2243] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:27.094484 18348 raft_consensus.cc:2272] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.096570 18348 ts_tablet_manager.cc:1916] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.098793 18223 tablet_bootstrap.cc:492] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:27.099268 18223 ts_tablet_manager.cc:1403] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.107s user 0.009s sys 0.004s
I20260924 19:57:27.099656 18118 raft_consensus.cc:2468] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:27.101720 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30a3acc4b104447885b27ea75dca1bea" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:27.101814 18223 raft_consensus.cc:359] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.102533 18223 raft_consensus.cc:385] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.102813 18223 raft_consensus.cc:740] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
W20260924 19:57:27.103781 17753 leader_election.cc:343] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: Tablet not RUNNING: STOPPED
I20260924 19:57:27.105719 18231 raft_consensus.cc:2243] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:27.105464 18223 consensus_queue.cc:260] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.106287 18231 raft_consensus.cc:2272] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.109272 18231 ts_tablet_manager.cc:1916] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.126004 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_c9486a2257a84d9ab489210722683584"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:27.128605 17607 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_3d0e38cde76d4300a129a04a04b5915d" } modify_external_catalogs: true
I20260924 19:57:27.129161 17607 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_3d0e38cde76d4300a129a04a04b5915d" } modify_external_catalogs: true
I20260924 19:57:27.134090 18097 tablet_service.cc:1511] Processing CreateTablet for tablet d81f74c7ba2443e49228b17e867588cc (DEFAULT_TABLE table=default.Table_e3a6abe9c34a4fd6a8567a9c0a8190c3 [id=4e80d69486a34b848393f01e656d122b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.135109 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d81f74c7ba2443e49228b17e867588cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.140635 17800 tablet_service.cc:1511] Processing CreateTablet for tablet d81f74c7ba2443e49228b17e867588cc (DEFAULT_TABLE table=default.Table_e3a6abe9c34a4fd6a8567a9c0a8190c3 [id=4e80d69486a34b848393f01e656d122b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.141665 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d81f74c7ba2443e49228b17e867588cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.142617 18301 ts_tablet_manager.cc:1929] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:27.142993 18301 log.cc:1199] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/30a3acc4b104447885b27ea75dca1bea
I20260924 19:57:27.143674 18301 ts_tablet_manager.cc:1950] T 30a3acc4b104447885b27ea75dca1bea P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
W20260924 19:57:27.144277 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9486a2257a84d9ab489210722683584 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:27.140142 18220 tablet_bootstrap.cc:492] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:27.147460 18220 ts_tablet_manager.cc:1403] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.186s user 0.008s sys 0.002s
I20260924 19:57:27.154023 18301 tablet_replica.cc:333] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:27.153980 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 30a3acc4b104447885b27ea75dca1bea (table default.Table_b013bc1fcd8b4e0db6c8d928b1984a3b [id=0fb7eef54311401c94d0c4e97de0cc0d]) successfully deleted
I20260924 19:57:27.159366 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet 08aa217e7ea34aa4b839d7e992f978eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:27.160748 18301 raft_consensus.cc:2243] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:27.161413 18301 raft_consensus.cc:2272] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.161957 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet 6edb7999719e4c3c93b93539bfe1550c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:27.163184 17950 tablet_service.cc:1511] Processing CreateTablet for tablet d81f74c7ba2443e49228b17e867588cc (DEFAULT_TABLE table=default.Table_e3a6abe9c34a4fd6a8567a9c0a8190c3 [id=4e80d69486a34b848393f01e656d122b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.163506 18301 ts_tablet_manager.cc:1916] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.164275 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d81f74c7ba2443e49228b17e867588cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.159416 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet c6f7f650fc26498faff2059b42bbf91f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:27.134294 18223 ts_tablet_manager.cc:1434] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20260924 19:57:27.168344 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet 30a3acc4b104447885b27ea75dca1bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:27.149820 18220 raft_consensus.cc:359] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:27.169598 18220 raft_consensus.cc:385] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.169885 18220 raft_consensus.cc:740] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.170856 17799 tablet_service.cc:1511] Processing CreateTablet for tablet a384c9a0bb65429ebb97fe0807c78773 (DEFAULT_TABLE table=default.Table_1a379ab4fc834d69865de7106e463200 [id=3ee2ed8b290747128a29f7c107dbfe82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.171988 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a384c9a0bb65429ebb97fe0807c78773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.175974 18220 consensus_queue.cc:260] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:27.181638 18098 tablet_service.cc:1511] Processing CreateTablet for tablet a384c9a0bb65429ebb97fe0807c78773 (DEFAULT_TABLE table=default.Table_1a379ab4fc834d69865de7106e463200 [id=3ee2ed8b290747128a29f7c107dbfe82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.182555 17606 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 6edb7999719e4c3c93b93539bfe1550c
I20260924 19:57:27.182901 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a384c9a0bb65429ebb97fe0807c78773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.186045 18223 tablet_bootstrap.cc:492] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:27.188274 17952 tablet_service.cc:1511] Processing CreateTablet for tablet a384c9a0bb65429ebb97fe0807c78773 (DEFAULT_TABLE table=default.Table_1a379ab4fc834d69865de7106e463200 [id=3ee2ed8b290747128a29f7c107dbfe82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.189260 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a384c9a0bb65429ebb97fe0807c78773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.194921 17610 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b701796a010f4e5a91d2314918867a9b" } new_table_name: "default.Table_8418d6fb70a647f896eb720d1958947c" modify_external_catalogs: true
I20260924 19:57:27.206702 18223 tablet_bootstrap.cc:654] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.209000 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_40bb698a81a44dc1ba9341b338e11772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:27.209372 18301 ts_tablet_manager.cc:1929] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260924 19:57:27.209717 18301 log.cc:1199] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/c6f7f650fc26498faff2059b42bbf91f
I20260924 19:57:27.210367 18301 ts_tablet_manager.cc:1950] T c6f7f650fc26498faff2059b42bbf91f P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
W20260924 19:57:27.210718 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40bb698a81a44dc1ba9341b338e11772 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:27.211337 17615 master_service.cc:960] ReplaceTablet: received request to replace tablet a9505fd426134d068b95639012112bf3 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:27.212680 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet c6f7f650fc26498faff2059b42bbf91f (table default.Table_f570a48d43e84d5d86ef5a1a92e3161c [id=c3b94a6ee3b5400cb352433d44a902f4]) successfully deleted
I20260924 19:57:27.213752 18220 ts_tablet_manager.cc:1434] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.066s user 0.005s sys 0.000s
I20260924 19:57:27.222034 17798 tablet_service.cc:1511] Processing CreateTablet for tablet 8a19b5812dc54589a5cbf48199951600 (DEFAULT_TABLE table=default.Table_a86d35d7e9de4a44a991798952b6de5c [id=06b8b073d6ae475c8125613b89463d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.223179 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a19b5812dc54589a5cbf48199951600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.235587 18301 tablet_replica.cc:333] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:27.239651 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_8e182a1f5e214229a76cb1a779e7203b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:27.240876 18095 tablet_service.cc:1511] Processing CreateTablet for tablet 8a19b5812dc54589a5cbf48199951600 (DEFAULT_TABLE table=default.Table_a86d35d7e9de4a44a991798952b6de5c [id=06b8b073d6ae475c8125613b89463d3c]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:57:27.241379 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e182a1f5e214229a76cb1a779e7203b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:27.242033 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a19b5812dc54589a5cbf48199951600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.243582 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 2 got 0
I20260924 19:57:27.248804 18096 tablet_service.cc:1558] Processing DeleteTablet for tablet 6edb7999719e4c3c93b93539bfe1550c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:27.248874 17951 tablet_service.cc:1558] Processing DeleteTablet for tablet 6edb7999719e4c3c93b93539bfe1550c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:27.249236 18220 tablet_bootstrap.cc:492] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:27.249356 18223 tablet_bootstrap.cc:492] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:27.249739 18223 ts_tablet_manager.cc:1403] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.002s
I20260924 19:57:27.248317 17606 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_a86d35d7e9de4a44a991798952b6de5c" } new_table_name: "default.Table_4b760755aa054287903e849789b54a61" modify_external_catalogs: true
I20260924 19:57:27.252478 18223 raft_consensus.cc:359] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.253090 18223 raft_consensus.cc:385] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.253329 18223 raft_consensus.cc:740] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.256827 17615 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet a9505fd426134d068b95639012112bf3
I20260924 19:57:27.258539 18301 raft_consensus.cc:2243] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:27.259310 18301 pending_rounds.cc:70] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: Trying to abort 1 pending ops.
I20260924 19:57:27.259575 18301 pending_rounds.cc:77] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986252882829312 op_type: NO_OP noop_request { }
I20260924 19:57:27.259501 18223 consensus_queue.cc:260] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.259980 18301 raft_consensus.cc:2889] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:27.260427 18301 raft_consensus.cc:2272] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.258217 17615 catalog_manager.cc:6934] ReplaceTablet: tablet a9505fd426134d068b95639012112bf3 deleted and replaced by tablet dd2f548d5bd5454cb16335473a9eeb03
I20260924 19:57:27.264796 17615 master_service.cc:960] ReplaceTablet: received request to replace tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:27.273156 17802 tablet_service.cc:1558] Processing DeleteTablet for tablet 6edb7999719e4c3c93b93539bfe1550c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:27.273911 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet a9505fd426134d068b95639012112bf3 with delete_type TABLET_DATA_DELETED (replaced by tablet dd2f548d5bd5454cb16335473a9eeb03) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:27.274088 17951 tablet_service.cc:1558] Processing DeleteTablet for tablet a9505fd426134d068b95639012112bf3 with delete_type TABLET_DATA_DELETED (replaced by tablet dd2f548d5bd5454cb16335473a9eeb03) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:27.278116 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 9b04059219084f608e8cb5fa4d0e0d58 (DEFAULT_TABLE table=default.Table_850acd445f004facb3a20aaac28003e3 [id=37ed98d304b9467fa4213115e9fd4da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.279253 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b04059219084f608e8cb5fa4d0e0d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.279587 18301 ts_tablet_manager.cc:1916] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.282408 18223 ts_tablet_manager.cc:1434] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.032s user 0.005s sys 0.002s
I20260924 19:57:27.296538 18223 tablet_bootstrap.cc:492] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:27.301543 18220 tablet_bootstrap.cc:654] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.307569 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet a9505fd426134d068b95639012112bf3 with delete_type TABLET_DATA_DELETED (replaced by tablet dd2f548d5bd5454cb16335473a9eeb03) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:27.309828 18223 tablet_bootstrap.cc:654] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.310699 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 9b04059219084f608e8cb5fa4d0e0d58 (DEFAULT_TABLE table=default.Table_850acd445f004facb3a20aaac28003e3 [id=37ed98d304b9467fa4213115e9fd4da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.311895 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b04059219084f608e8cb5fa4d0e0d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.312618 18301 ts_tablet_manager.cc:1929] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:27.319319 18301 log.cc:1199] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/90cd09aecb6247e7b7937996f60f007f
I20260924 19:57:27.320147 18301 ts_tablet_manager.cc:1950] T 90cd09aecb6247e7b7937996f60f007f P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:27.323379 18301 tablet_replica.cc:333] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:27.282975 17607 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet fb011ae3f182464ab68d77924ab70181
I20260924 19:57:27.328524 18223 tablet_bootstrap.cc:492] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:27.331470 18223 ts_tablet_manager.cc:1403] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.001s
I20260924 19:57:27.331898 18301 raft_consensus.cc:2243] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:27.332448 18301 raft_consensus.cc:2272] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.334084 18223 raft_consensus.cc:359] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.334695 18223 raft_consensus.cc:385] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.335218 18223 raft_consensus.cc:740] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.336071 17802 tablet_service.cc:1511] Processing CreateTablet for tablet f90a677e94034c73ab5452554862c1c3 (DEFAULT_TABLE table=default.Table_fca275d61c914e72b792ca0bb116d0bf [id=19596b2bff2e46399e96dcc6d5fbfbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.307749 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_ae0e486aaf984e31bdbc62e74fbff121"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:27.322669 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 90cd09aecb6247e7b7937996f60f007f (table default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]) successfully deleted
I20260924 19:57:27.337235 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f90a677e94034c73ab5452554862c1c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:27.338174 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae0e486aaf984e31bdbc62e74fbff121 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:27.338696 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet fb011ae3f182464ab68d77924ab70181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:27 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:27.337047 18301 ts_tablet_manager.cc:1916] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.342244 18223 consensus_queue.cc:260] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.348332 18231 ts_tablet_manager.cc:1929] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:27.348747 18231 log.cc:1199] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/90cd09aecb6247e7b7937996f60f007f
I20260924 19:57:27.349438 18231 ts_tablet_manager.cc:1950] T 90cd09aecb6247e7b7937996f60f007f P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:27.352021 18231 tablet_replica.cc:333] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:27.352246 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 90cd09aecb6247e7b7937996f60f007f (table default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]) successfully deleted
I20260924 19:57:27.355502 17613 catalog_manager.cc:5838] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:27.357146 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet f2897faafbff460bae99990a5940d38b (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:27.362885 18223 ts_tablet_manager.cc:1434] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.031s user 0.005s sys 0.002s
I20260924 19:57:27.365211 17947 tablet_service.cc:1511] Processing CreateTablet for tablet f90a677e94034c73ab5452554862c1c3 (DEFAULT_TABLE table=default.Table_fca275d61c914e72b792ca0bb116d0bf [id=19596b2bff2e46399e96dcc6d5fbfbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.366415 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f90a677e94034c73ab5452554862c1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.367010 18223 tablet_bootstrap.cc:492] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:27.303948 17951 tablet_service.cc:1511] Processing CreateTablet for tablet 8a19b5812dc54589a5cbf48199951600 (DEFAULT_TABLE table=default.Table_a86d35d7e9de4a44a991798952b6de5c [id=06b8b073d6ae475c8125613b89463d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.370492 18095 tablet_service.cc:1511] Processing CreateTablet for tablet f90a677e94034c73ab5452554862c1c3 (DEFAULT_TABLE table=default.Table_fca275d61c914e72b792ca0bb116d0bf [id=19596b2bff2e46399e96dcc6d5fbfbed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.318790 17949 tablet_service.cc:1511] Processing CreateTablet for tablet 9b04059219084f608e8cb5fa4d0e0d58 (DEFAULT_TABLE table=default.Table_850acd445f004facb3a20aaac28003e3 [id=37ed98d304b9467fa4213115e9fd4da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:27.339406 17948 tablet_service.cc:1558] Processing DeleteTablet for tablet fb011ae3f182464ab68d77924ab70181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:27 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:27.371570 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f90a677e94034c73ab5452554862c1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.371686 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b04059219084f608e8cb5fa4d0e0d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.375802 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a19b5812dc54589a5cbf48199951600. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:27.385965 18223 tablet_bootstrap.cc:654] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:27.390168 17610 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83ee43b6dc064dc7b320691d52e57ea7 consensus state has no leader
I20260924 19:57:27.396867 18220 tablet_bootstrap.cc:492] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:27.397292 18220 ts_tablet_manager.cc:1403] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.148s user 0.004s sys 0.004s
I20260924 19:57:27.398927 18348 ts_tablet_manager.cc:1929] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260924 19:57:27.404258 18348 log.cc:1199] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/c6f7f650fc26498faff2059b42bbf91f
I20260924 19:57:27.406683 18348 ts_tablet_manager.cc:1950] T c6f7f650fc26498faff2059b42bbf91f P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:27.412176 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet c6f7f650fc26498faff2059b42bbf91f (table default.Table_f570a48d43e84d5d86ef5a1a92e3161c [id=c3b94a6ee3b5400cb352433d44a902f4]) successfully deleted
I20260924 19:57:27.416877 18223 tablet_bootstrap.cc:492] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:27.417269 18223 ts_tablet_manager.cc:1403] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.004s
I20260924 19:57:27.418888 18301 ts_tablet_manager.cc:1929] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:27.419255 18301 log.cc:1199] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/6edb7999719e4c3c93b93539bfe1550c
I20260924 19:57:27.408735 18220 raft_consensus.cc:359] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.419703 18220 raft_consensus.cc:385] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.419966 18301 ts_tablet_manager.cc:1950] T 6edb7999719e4c3c93b93539bfe1550c P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:27.420439 18220 raft_consensus.cc:740] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.421415 18223 raft_consensus.cc:359] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.422205 18223 raft_consensus.cc:385] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.422470 18223 raft_consensus.cc:740] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.422482 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 6edb7999719e4c3c93b93539bfe1550c (table default.Table_88bf80a74c3245d1a74b3757ef4eae71 [id=ecd769593d2d4b86a333ebf13c174a24]) successfully deleted
W20260924 19:57:27.423144 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet a9505fd426134d068b95639012112bf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9505fd426134d068b95639012112bf3 already in progress: creating tablet
I20260924 19:57:27.423117 18223 consensus_queue.cc:260] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.424273 18348 tablet_replica.cc:333] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:27.427094 18348 raft_consensus.cc:2243] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:27.432782 18301 tablet_replica.cc:333] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:27.441885 18301 raft_consensus.cc:2243] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:27.442031 18223 ts_tablet_manager.cc:1434] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260924 19:57:27.442354 18301 raft_consensus.cc:2272] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.424844 18220 consensus_queue.cc:260] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.445673 18301 ts_tablet_manager.cc:1916] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.447445 18223 tablet_bootstrap.cc:492] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:27.452291 17611 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_8e182a1f5e214229a76cb1a779e7203b" } modify_external_catalogs: true
I20260924 19:57:27.452880 17611 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_8e182a1f5e214229a76cb1a779e7203b" } modify_external_catalogs: true
I20260924 19:57:27.458783 18223 tablet_bootstrap.cc:654] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.459965 17612 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_8418d6fb70a647f896eb720d1958947c [id=31e1c3788d264823a12ecd9755ec2027]). Expected version 3 got 0
I20260924 19:57:27.464052 18220 ts_tablet_manager.cc:1434] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.066s user 0.005s sys 0.002s
I20260924 19:57:27.442517 18348 raft_consensus.cc:2272] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.466249 18348 ts_tablet_manager.cc:1916] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.433172 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet fb011ae3f182464ab68d77924ab70181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:27 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:27.481923 17612 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_8418d6fb70a647f896eb720d1958947c [id=31e1c3788d264823a12ecd9755ec2027]). Expected version 3 got 0
I20260924 19:57:27.435959 18224 tablet_bootstrap.cc:492] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:27.483449 18224 ts_tablet_manager.cc:1403] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.532s user 0.014s sys 0.015s
I20260924 19:57:27.484257 18223 tablet_bootstrap.cc:492] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:27.484663 18223 ts_tablet_manager.cc:1403] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.000s
I20260924 19:57:27.485904 18224 raft_consensus.cc:359] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:27.486523 18224 raft_consensus.cc:385] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.486784 18224 raft_consensus.cc:740] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.487110 18223 raft_consensus.cc:359] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.487581 17610 master_service.cc:960] ReplaceTablet: received request to replace tablet f2897faafbff460bae99990a5940d38b from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:27.487815 18223 raft_consensus.cc:385] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.488083 18223 raft_consensus.cc:740] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.488933 18223 consensus_queue.cc:260] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.490276 18231 raft_consensus.cc:2243] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:27.495950 18231 raft_consensus.cc:2272] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.496855 18220 tablet_bootstrap.cc:492] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:27.496946 18301 ts_tablet_manager.cc:1929] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:27.497284 18301 log.cc:1199] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/fb011ae3f182464ab68d77924ab70181
I20260924 19:57:27.497870 18301 ts_tablet_manager.cc:1950] T fb011ae3f182464ab68d77924ab70181 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:27.499948 18231 ts_tablet_manager.cc:1916] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.500250 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet fb011ae3f182464ab68d77924ab70181 (table default.Table_3d0e38cde76d4300a129a04a04b5915d [id=2d056489f9314bb59c83e506d47ca5b8]) successfully deleted
I20260924 19:57:27.502656 18220 tablet_bootstrap.cc:654] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.505579 18223 ts_tablet_manager.cc:1434] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260924 19:57:27.509871 18223 tablet_bootstrap.cc:492] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:27.517175 17606 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a19b5812dc54589a5cbf48199951600 consensus state has no leader
I20260924 19:57:27.520778 17609 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_14bb83c1f8ee444d81a998ed429a7b60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:27.522591 17609 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14bb83c1f8ee444d81a998ed429a7b60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:27.529851 17606 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_4a15e8e61f934d08b802fd02d421cd3f" } modify_external_catalogs: true
I20260924 19:57:27.530452 17606 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_4a15e8e61f934d08b802fd02d421cd3f" } modify_external_catalogs: true
I20260924 19:57:27.531688 18223 tablet_bootstrap.cc:654] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.538231 18361 raft_consensus.cc:493] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:27.540004 18361 raft_consensus.cc:515] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.543465 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ee43b6dc064dc7b320691d52e57ea7" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:27.544090 17972 raft_consensus.cc:2468] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:27.545684 18050 leader_election.cc:304] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:27.547490 18362 raft_consensus.cc:2804] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:27.547840 18362 raft_consensus.cc:493] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:27.548240 18362 raft_consensus.cc:3060] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.562539 18364 raft_consensus.cc:493] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:27.563127 18364 raft_consensus.cc:515] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:27.566283 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8cc802bf87f4d739175f93203478af8" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:27.566915 17972 raft_consensus.cc:2468] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:27.568251 18050 leader_election.cc:304] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:27.569172 18322 raft_consensus.cc:2804] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:27.569512 18322 raft_consensus.cc:493] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:27.569865 18322 raft_consensus.cc:3060] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.572423 18220 tablet_bootstrap.cc:492] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:27.572831 18220 ts_tablet_manager.cc:1403] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.003s
I20260924 19:57:27.575361 18220 raft_consensus.cc:359] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.575759 18223 tablet_bootstrap.cc:492] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:27.576005 18220 raft_consensus.cc:385] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.576190 18223 ts_tablet_manager.cc:1403] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.003s
I20260924 19:57:27.576262 18220 raft_consensus.cc:740] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.578477 18223 raft_consensus.cc:359] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:27.579706 18364 leader_election.cc:290] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:27.579064 18223 raft_consensus.cc:385] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.580317 18223 raft_consensus.cc:740] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.580900 18223 consensus_queue.cc:260] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:27.582667 18223 ts_tablet_manager.cc:1434] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 19:57:27.583410 18220 consensus_queue.cc:260] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.600529 18322 raft_consensus.cc:515] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:27.601751 18220 ts_tablet_manager.cc:1434] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260924 19:57:27.603250 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8cc802bf87f4d739175f93203478af8" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:27.603817 17972 raft_consensus.cc:3060] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.611912 17972 raft_consensus.cc:2468] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:27.613130 18050 leader_election.cc:304] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:27.614661 18364 raft_consensus.cc:2804] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:27.615100 18364 raft_consensus.cc:697] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:27.615931 18364 consensus_queue.cc:237] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:27.618295 18223 tablet_bootstrap.cc:492] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:27.620138 18362 raft_consensus.cc:515] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.622938 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ee43b6dc064dc7b320691d52e57ea7" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:27.623548 17972 raft_consensus.cc:3060] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.628281 18348 ts_tablet_manager.cc:1929] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:27.628693 18348 log.cc:1199] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/90cd09aecb6247e7b7937996f60f007f
I20260924 19:57:27.629369 18348 ts_tablet_manager.cc:1950] T 90cd09aecb6247e7b7937996f60f007f P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:27.628159 18322 leader_election.cc:290] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:27.511219 18224 consensus_queue.cc:260] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:27.596299 17753 consensus_peers.cc:597] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f33d435d7a594344956a88506a43ef59. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:27.637835 18224 ts_tablet_manager.cc:1434] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.154s user 0.005s sys 0.000s
I20260924 19:57:27.638847 18224 tablet_bootstrap.cc:492] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:27.647897 18224 tablet_bootstrap.cc:654] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.652284 18220 tablet_bootstrap.cc:492] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
W20260924 19:57:27.652364 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 75172us
I20260924 19:57:27.654192 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8cc802bf87f4d739175f93203478af8" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:27.654868 17821 raft_consensus.cc:2468] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:27.658600 17612 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_4b760755aa054287903e849789b54a61" } new_table_name: "default.Table_37482fb26ba144619bb66861dceaab3b" modify_external_catalogs: true
I20260924 19:57:27.659077 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:27.661067 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:27.663494 18223 tablet_bootstrap.cc:654] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.668735 18361 leader_election.cc:290] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:27.670958 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 90cd09aecb6247e7b7937996f60f007f (table default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]) successfully deleted
I20260924 19:57:27.672067 18348 tablet_replica.cc:333] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:27.673056 18348 raft_consensus.cc:2243] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:27.688519 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8cc802bf87f4d739175f93203478af8" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:27.689173 17821 raft_consensus.cc:3060] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.691548 18223 tablet_bootstrap.cc:492] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:27.691979 18223 ts_tablet_manager.cc:1403] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.000s
I20260924 19:57:27.705125 17821 raft_consensus.cc:2468] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:27.707047 18223 raft_consensus.cc:359] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:27.707726 18223 raft_consensus.cc:385] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.707973 18223 raft_consensus.cc:740] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.708664 18348 raft_consensus.cc:2272] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.708509 18223 consensus_queue.cc:260] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:27.710306 18223 ts_tablet_manager.cc:1434] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20260924 19:57:27.713717 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ee43b6dc064dc7b320691d52e57ea7" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:27.718174 18223 tablet_bootstrap.cc:492] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:27.723368 18223 tablet_bootstrap.cc:654] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:27.724534 18051 leader_election.cc:343] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:27.708052 18220 tablet_bootstrap.cc:654] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.733741 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet 30a3acc4b104447885b27ea75dca1bea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:26 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:27.734969 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet f33d435d7a594344956a88506a43ef59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:25 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:27.736095 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet 90cd09aecb6247e7b7937996f60f007f with delete_type TABLET_DATA_DELETED (replaced by tablet efd7339d0fee436299a7eac02a82addd) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:27.739909 18348 ts_tablet_manager.cc:1916] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.742084 18224 tablet_bootstrap.cc:492] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:27.742580 18224 ts_tablet_manager.cc:1403] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.104s user 0.010s sys 0.003s
I20260924 19:57:27.744174 17611 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet e0a4b58fd6704ed4a1c782269b53d188
I20260924 19:57:27.745226 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_a4745efada6f4edea3f2d4bad1c3905e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:27.745412 18224 raft_consensus.cc:359] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.746313 18224 raft_consensus.cc:385] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.746644 18224 raft_consensus.cc:740] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
W20260924 19:57:27.747370 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4745efada6f4edea3f2d4bad1c3905e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:27.747468 18224 consensus_queue.cc:260] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.750646 18372 raft_consensus.cc:493] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:27.751200 18372 raft_consensus.cc:515] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.753247 18372 leader_election.cc:290] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:27.756820 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1d456a7ccbe4f03b41d6b52ae382424" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:27.757483 18118 raft_consensus.cc:2468] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:27.759580 17753 leader_election.cc:304] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:27.759716 18371 raft_consensus.cc:493] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:27.759536 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ee43b6dc064dc7b320691d52e57ea7" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:27.760183 18371 raft_consensus.cc:515] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:27.760555 18372 raft_consensus.cc:2804] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:27.738686 18362 leader_election.cc:290] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:27.761080 18372 raft_consensus.cc:493] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:27.761189 18362 raft_consensus.cc:493] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:27.761543 18372 raft_consensus.cc:3060] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.761649 18362 raft_consensus.cc:515] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.762351 18371 leader_election.cc:290] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:27.763870 18362 leader_election.cc:290] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:27.765030 18050 leader_election.cc:343] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Not found: Tablet not found: 8a19b5812dc54589a5cbf48199951600
I20260924 19:57:27.774724 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a19b5812dc54589a5cbf48199951600" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:27.781162 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9505fd426134d068b95639012112bf3" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:27.756028 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1d456a7ccbe4f03b41d6b52ae382424" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:27.782795 17970 raft_consensus.cc:2468] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:27.766317 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9505fd426134d068b95639012112bf3" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:27.784799 18051 leader_election.cc:343] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:27.771430 17972 raft_consensus.cc:2468] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:27.792743 17609 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_96afe0911af349bf9301ce80ef227bd0" } new_table_name: "default.Table_b2d71701938e43c2bb30b8594cb20cf0" modify_external_catalogs: true
W20260924 19:57:27.797696 17602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.190:46309, user_credentials={real_user=slave}} blocked reactor thread for 65843.4us
W20260924 19:57:27.785148 18050 leader_election.cc:343] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:27.792779 18223 tablet_bootstrap.cc:492] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:27.806857 18223 ts_tablet_manager.cc:1403] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.091s user 0.011s sys 0.024s
I20260924 19:57:27.808966 18050 leader_election.cc:304] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
I20260924 19:57:27.809386 18223 raft_consensus.cc:359] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:27.810005 18361 raft_consensus.cc:2804] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:27.810034 18223 raft_consensus.cc:385] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.810465 18223 raft_consensus.cc:740] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.810581 18361 raft_consensus.cc:697] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:27.811189 18223 consensus_queue.cc:260] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:27.811493 18361 consensus_queue.cc:237] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.815706 18223 ts_tablet_manager.cc:1434] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20260924 19:57:27.816550 18051 leader_election.cc:343] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:27.817580 18372 raft_consensus.cc:515] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.818554 18223 tablet_bootstrap.cc:492] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:27.818424 18051 leader_election.cc:304] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:27.821830 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1d456a7ccbe4f03b41d6b52ae382424" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:27.822532 17972 raft_consensus.cc:3060] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.823849 18223 tablet_bootstrap.cc:654] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:27.823977 18051 leader_election.cc:343] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:27.824431 18051 leader_election.cc:304] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:27.826718 18224 ts_tablet_manager.cc:1434] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.084s user 0.002s sys 0.004s
I20260924 19:57:27.830981 18371 raft_consensus.cc:2749] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:27.831629 18371 raft_consensus.cc:2749] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:27.836979 18220 tablet_bootstrap.cc:492] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:27.845011 18223 tablet_bootstrap.cc:492] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:27.845433 18223 ts_tablet_manager.cc:1403] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20260924 19:57:27.845645 18220 ts_tablet_manager.cc:1403] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.194s user 0.011s sys 0.000s
W20260924 19:57:27.829999 17753 consensus_peers.cc:597] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6edb7999719e4c3c93b93539bfe1550c. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:27.850420 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f1d456a7ccbe4f03b41d6b52ae382424" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:27.851034 18118 raft_consensus.cc:3060] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.858410 18118 raft_consensus.cc:2468] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:27.843740 18372 leader_election.cc:290] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:27.849926 18224 tablet_bootstrap.cc:492] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:27.870134 17606 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet f1d456a7ccbe4f03b41d6b52ae382424
I20260924 19:57:27.877817 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet f1d456a7ccbe4f03b41d6b52ae382424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:27 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:27.860011 18223 raft_consensus.cc:359] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:27.882154 18223 raft_consensus.cc:385] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.882455 18223 raft_consensus.cc:740] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.873268 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet f1d456a7ccbe4f03b41d6b52ae382424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:27 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:27.883069 17753 leader_election.cc:304] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:27.883678 18301 tablet_replica.cc:333] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:27.884408 18301 raft_consensus.cc:2243] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:27.884303 18372 raft_consensus.cc:2804] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:27.884836 18301 raft_consensus.cc:2272] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.883086 18223 consensus_queue.cc:260] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:27.888674 18231 ts_tablet_manager.cc:1929] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:27.889063 18231 log.cc:1199] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/08aa217e7ea34aa4b839d7e992f978eb
I20260924 19:57:27.889176 18372 raft_consensus.cc:697] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:27.891512 18301 ts_tablet_manager.cc:1916] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.876730 17951 tablet_service.cc:1558] Processing DeleteTablet for tablet f1d456a7ccbe4f03b41d6b52ae382424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:27 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:27.848366 18220 raft_consensus.cc:359] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.890479 18372 consensus_queue.cc:237] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.894845 18220 raft_consensus.cc:385] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.890923 17972 raft_consensus.cc:2468] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:27.896236 18220 raft_consensus.cc:740] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.896590 18223 ts_tablet_manager.cc:1434] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.051s user 0.008s sys 0.000s
I20260924 19:57:27.897452 18223 tablet_bootstrap.cc:492] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:27.900842 18220 consensus_queue.cc:260] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.907222 18224 tablet_bootstrap.cc:654] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:27.907833 17612 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a19b5812dc54589a5cbf48199951600 consensus state has no leader
I20260924 19:57:27.912704 18361 raft_consensus.cc:493] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:27.913200 18361 raft_consensus.cc:515] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:27.915885 18361 leader_election.cc:290] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:27.916119 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b9ffd3fc8714979bfaa24c36d6fe0f9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:27.917127 17972 raft_consensus.cc:2468] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:27.917642 18223 tablet_bootstrap.cc:654] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:27.917606 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b9ffd3fc8714979bfaa24c36d6fe0f9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:27.918763 18220 ts_tablet_manager.cc:1434] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.073s user 0.007s sys 0.000s
I20260924 19:57:27.889755 18231 ts_tablet_manager.cc:1950] T 08aa217e7ea34aa4b839d7e992f978eb P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:27.919379 18050 leader_election.cc:304] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
W20260924 19:57:27.920493 18051 leader_election.cc:343] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:27.921391 18361 raft_consensus.cc:2804] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:27.921772 18361 raft_consensus.cc:493] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:27.922122 18361 raft_consensus.cc:3060] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.924090 18220 tablet_bootstrap.cc:492] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:27.926252 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 08aa217e7ea34aa4b839d7e992f978eb (table default.Table_87b845fa3a064c269030287021ae1c05 [id=617d557c676d462b9bf9c44c4d5310aa]) successfully deleted
I20260924 19:57:27.929782 18361 raft_consensus.cc:515] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:27.930680 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet c6f7f650fc26498faff2059b42bbf91f because the tablet was not found. No further retry: Not found: Tablet not found: c6f7f650fc26498faff2059b42bbf91f
I20260924 19:57:27.933327 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_c2dba817ef2249bd8c401ea137c347e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:27.935094 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2dba817ef2249bd8c401ea137c347e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:27.937840 18231 tablet_replica.cc:333] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
W20260924 19:57:27.940472 17602 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 8a19b5812dc54589a5cbf48199951600 (table default.Table_37482fb26ba144619bb66861dceaab3b [id=06b8b073d6ae475c8125613b89463d3c], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8a19b5812dc54589a5cbf48199951600 consensus state has no leader
I20260924 19:57:27.940912 18223 tablet_bootstrap.cc:492] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:27.941308 18223 ts_tablet_manager.cc:1403] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.001s
I20260924 19:57:27.943718 18223 raft_consensus.cc:359] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:27.944343 18223 raft_consensus.cc:385] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:27.944592 18223 raft_consensus.cc:740] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:27.947381 18231 raft_consensus.cc:2243] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:27.952935 18301 ts_tablet_manager.cc:1929] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:27.953305 18301 log.cc:1199] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/f1d456a7ccbe4f03b41d6b52ae382424
I20260924 19:57:27.959704 18301 ts_tablet_manager.cc:1950] T f1d456a7ccbe4f03b41d6b52ae382424 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:27.962174 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet f1d456a7ccbe4f03b41d6b52ae382424 (table default.Table_4a15e8e61f934d08b802fd02d421cd3f [id=ef215e1163d74af69a3a3f6f763c3249]) successfully deleted
I20260924 19:57:27.962925 18223 consensus_queue.cc:260] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:27.967095 18223 ts_tablet_manager.cc:1434] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260924 19:57:27.967794 18290 consensus_queue.cc:1048] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:27.975296 18348 ts_tablet_manager.cc:1929] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:27.975739 18348 log.cc:1199] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/6edb7999719e4c3c93b93539bfe1550c
I20260924 19:57:27.953042 18231 raft_consensus.cc:2272] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.987593 18231 ts_tablet_manager.cc:1916] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:27.993809 17610 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet f2897faafbff460bae99990a5940d38b
I20260924 19:57:27.995715 17610 catalog_manager.cc:6934] ReplaceTablet: tablet f2897faafbff460bae99990a5940d38b deleted and replaced by tablet 8a524bad1c4040e58ad84fcf5a4fc47d
I20260924 19:57:27.996124 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet f2897faafbff460bae99990a5940d38b with delete_type TABLET_DATA_DELETED (replaced by tablet 8a524bad1c4040e58ad84fcf5a4fc47d) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:27.996778 18301 tablet_replica.cc:333] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:27.997524 18301 raft_consensus.cc:2243] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:27.997960 18301 raft_consensus.cc:2272] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:27.999653 18301 ts_tablet_manager.cc:1916] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:28.000002 17951 tablet_service.cc:1558] Processing DeleteTablet for tablet f2897faafbff460bae99990a5940d38b with delete_type TABLET_DATA_DELETED (replaced by tablet 8a524bad1c4040e58ad84fcf5a4fc47d) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:27.998879 17608 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 2 got 0
I20260924 19:57:28.002849 17610 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_ae0e486aaf984e31bdbc62e74fbff121" } modify_external_catalogs: true
I20260924 19:57:28.003536 17610 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_ae0e486aaf984e31bdbc62e74fbff121" } modify_external_catalogs: true
I20260924 19:57:28.005604 18348 ts_tablet_manager.cc:1950] T 6edb7999719e4c3c93b93539bfe1550c P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:28.006215 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_412220a60c3e4a16a5c4abcbbc48357c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:28.006886 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 2 got 0
W20260924 19:57:28.008061 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_412220a60c3e4a16a5c4abcbbc48357c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:28.011111 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b9ffd3fc8714979bfaa24c36d6fe0f9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:28.015413 17972 raft_consensus.cc:3060] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:27.992939 18220 tablet_bootstrap.cc:654] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:27.990899 17753 consensus_peers.cc:597] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f33d435d7a594344956a88506a43ef59. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:28.025529 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b9ffd3fc8714979bfaa24c36d6fe0f9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
W20260924 19:57:28.026166 17753 consensus_peers.cc:597] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:28.026558 17970 raft_consensus.cc:1275] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Refusing update from remote peer c4f6e3a7b06148918d230c77ab0c8202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 19:57:28.026958 18051 leader_election.cc:343] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:28.020306 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet f2897faafbff460bae99990a5940d38b with delete_type TABLET_DATA_DELETED (replaced by tablet 8a524bad1c4040e58ad84fcf5a4fc47d) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:28.027912 18361 leader_election.cc:290] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:28.028399 18372 consensus_queue.cc:1048] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:28.029314 17972 raft_consensus.cc:2468] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:28.030565 18050 leader_election.cc:304] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
I20260924 19:57:28.031767 18361 raft_consensus.cc:2804] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:28.034691 18224 tablet_bootstrap.cc:492] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:28.035128 18224 ts_tablet_manager.cc:1403] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.185s user 0.011s sys 0.001s
I20260924 19:57:28.037078 18301 ts_tablet_manager.cc:1929] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:28.037523 18301 log.cc:1199] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/f2897faafbff460bae99990a5940d38b
I20260924 19:57:28.038162 18301 ts_tablet_manager.cc:1950] T f2897faafbff460bae99990a5940d38b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:28.037741 18224 raft_consensus.cc:359] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.038417 18224 raft_consensus.cc:385] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.044782 18361 raft_consensus.cc:697] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:28.053072 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet f2897faafbff460bae99990a5940d38b (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]) successfully deleted
I20260924 19:57:28.054841 18348 tablet_replica.cc:333] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:28.055727 18348 raft_consensus.cc:2243] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 19:57:28.058015 17753 consensus_peers.cc:597] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30a3acc4b104447885b27ea75dca1bea. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:28.057025 18361 consensus_queue.cc:237] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.056226 18348 pending_rounds.cc:70] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: Trying to abort 1 pending ops.
I20260924 19:57:28.038703 18224 raft_consensus.cc:740] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
W20260924 19:57:28.062831 17753 consensus_peers.cc:597] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:28.061058 18348 pending_rounds.cc:77] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986253681160192 op_type: NO_OP noop_request { }
I20260924 19:57:28.063428 18348 raft_consensus.cc:2889] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:28.063896 18348 raft_consensus.cc:2272] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:28.064724 18224 consensus_queue.cc:260] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.073278 18224 ts_tablet_manager.cc:1434] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.038s user 0.002s sys 0.003s
I20260924 19:57:28.074618 17606 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_8418d6fb70a647f896eb720d1958947c [id=31e1c3788d264823a12ecd9755ec2027]). Expected version 3 got 0
I20260924 19:57:28.083684 17606 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_40bb698a81a44dc1ba9341b338e11772" } modify_external_catalogs: true
I20260924 19:57:28.084306 17606 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_40bb698a81a44dc1ba9341b338e11772" } modify_external_catalogs: true
I20260924 19:57:28.099967 18348 ts_tablet_manager.cc:1916] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:28.105180 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_8418d6fb70a647f896eb720d1958947c [id=31e1c3788d264823a12ecd9755ec2027]). Expected version 3 got 0
I20260924 19:57:28.106083 17615 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9
I20260924 19:57:28.107692 17615 catalog_manager.cc:6934] ReplaceTablet: tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 deleted and replaced by tablet 954178b8cc1e434085337f66b0e85672
I20260924 19:57:28.116765 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 954178b8cc1e434085337f66b0e85672) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:28.117467 18301 tablet_replica.cc:333] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:28.122190 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 954178b8cc1e434085337f66b0e85672) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:28.130388 18391 raft_consensus.cc:493] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:28.130900 18391 raft_consensus.cc:515] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.133240 18391 leader_election.cc:290] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:28.138041 17610 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet b64bada35f8248bfaa852358f3f4d25b
I20260924 19:57:28.141605 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_cca06980f109487a9ed8b8172950b60d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:28.143441 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cca06980f109487a9ed8b8172950b60d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:28.144743 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ee43b6dc064dc7b320691d52e57ea7" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:28.115402 18220 tablet_bootstrap.cc:492] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:28.117434 17951 tablet_service.cc:1558] Processing DeleteTablet for tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 954178b8cc1e434085337f66b0e85672) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:28.155635 18220 ts_tablet_manager.cc:1403] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.232s user 0.010s sys 0.001s
I20260924 19:57:28.113200 18394 tablet.cc:1751] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:28.146431 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ee43b6dc064dc7b320691d52e57ea7" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:28.157450 17970 raft_consensus.cc:2393] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c4f6e3a7b06148918d230c77ab0c8202 in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:28.158447 18220 raft_consensus.cc:359] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.159201 18220 raft_consensus.cc:385] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.159498 18220 raft_consensus.cc:740] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:28.160323 18220 consensus_queue.cc:260] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.157998 18393 tablet.cc:1751] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:28.159533 17753 leader_election.cc:304] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:28.163448 18372 raft_consensus.cc:3060] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:28.171623 18372 raft_consensus.cc:2749] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:28.158180 18224 tablet_bootstrap.cc:492] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:28.178669 18301 raft_consensus.cc:2243] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:28.181494 18301 pending_rounds.cc:70] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: Trying to abort 1 pending ops.
I20260924 19:57:28.181730 18301 pending_rounds.cc:77] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986257677672448 op_type: NO_OP noop_request { }
I20260924 19:57:28.182134 18220 ts_tablet_manager.cc:1434] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260924 19:57:28.182065 18301 raft_consensus.cc:2889] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:28.182740 18301 raft_consensus.cc:2272] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:28.199574 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_7190cd8a957a47368155eba08e1e37f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:28.200487 18220 tablet_bootstrap.cc:492] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:28.216773 18362 raft_consensus.cc:493] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:28.217226 18362 raft_consensus.cc:515] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.219314 18231 ts_tablet_manager.cc:1929] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:28.219756 18231 log.cc:1199] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/6edb7999719e4c3c93b93539bfe1550c
I20260924 19:57:28.220572 18231 ts_tablet_manager.cc:1950] T 6edb7999719e4c3c93b93539bfe1550c P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:28.221868 18220 tablet_bootstrap.cc:654] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:28.219110 18362 leader_election.cc:290] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:28.225719 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81f74c7ba2443e49228b17e867588cc" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
W20260924 19:57:28.227322 18051 leader_election.cc:343] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:28.224462 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81f74c7ba2443e49228b17e867588cc" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:28.229943 18050 leader_election.cc:343] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:28.230412 18050 leader_election.cc:304] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:28.231609 18361 raft_consensus.cc:2749] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:28.176874 17613 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_8418d6fb70a647f896eb720d1958947c" } new_table_name: "default.Table_7c382a08aec546da8c1aa720956036c7" modify_external_catalogs: true
I20260924 19:57:28.243607 17606 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 3c12af80e341422b9d38bfc972f29956
W20260924 19:57:28.250152 18392 consensus_queue.cc:1175] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
W20260924 19:57:28.251235 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7190cd8a957a47368155eba08e1e37f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:28.253515 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 6edb7999719e4c3c93b93539bfe1550c (table default.Table_88bf80a74c3245d1a74b3757ef4eae71 [id=ecd769593d2d4b86a333ebf13c174a24]) successfully deleted
W20260924 19:57:28.254276 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet a9505fd426134d068b95639012112bf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9505fd426134d068b95639012112bf3 already in progress: creating tablet
I20260924 19:57:28.255573 18231 tablet_replica.cc:333] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:28.255645 18301 ts_tablet_manager.cc:1916] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:28.260521 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_2be07cf4cdd64ac5a99db4037ee35542"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:28.264896 18220 tablet_bootstrap.cc:492] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:28.265313 18220 ts_tablet_manager.cc:1403] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.002s
W20260924 19:57:28.266438 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet fb011ae3f182464ab68d77924ab70181 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb011ae3f182464ab68d77924ab70181 already in progress: creating tablet
I20260924 19:57:28.267998 18220 raft_consensus.cc:359] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.268641 18220 raft_consensus.cc:385] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.268909 18220 raft_consensus.cc:740] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
W20260924 19:57:28.276067 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2be07cf4cdd64ac5a99db4037ee35542 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:28.276472 18231 raft_consensus.cc:2243] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:28.277411 18231 raft_consensus.cc:2272] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:28.279778 18231 ts_tablet_manager.cc:1916] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:28.284323 17607 master_service.cc:960] ReplaceTablet: received request to replace tablet fb011ae3f182464ab68d77924ab70181 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:28.288795 18224 tablet_bootstrap.cc:654] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:28.291461 18220 consensus_queue.cc:260] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.298990 18220 ts_tablet_manager.cc:1434] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20260924 19:57:28.308494 18220 tablet_bootstrap.cc:492] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:28.326802 18220 tablet_bootstrap.cc:654] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:28.333003 17607 master_service.cc:960] ReplaceTablet: received request to replace tablet 8a19b5812dc54589a5cbf48199951600 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:28.335620 18224 tablet_bootstrap.cc:492] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:28.336094 18224 ts_tablet_manager.cc:1403] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.262s user 0.006s sys 0.004s
I20260924 19:57:28.338445 18224 raft_consensus.cc:359] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.339264 18224 raft_consensus.cc:385] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.339589 18224 raft_consensus.cc:740] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:28.342464 17612 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_a07eb85445714c59bb9837bc7228144c" } modify_external_catalogs: true
I20260924 19:57:28.343083 17612 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_a07eb85445714c59bb9837bc7228144c" } modify_external_catalogs: true
I20260924 19:57:28.353538 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_c8e27940040c495780f92f92431541cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:28.345633 18231 ts_tablet_manager.cc:1929] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:28.355336 18231 log.cc:1199] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/30a3acc4b104447885b27ea75dca1bea
I20260924 19:57:28.356073 18231 ts_tablet_manager.cc:1950] T 30a3acc4b104447885b27ea75dca1bea P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:28.358759 18301 ts_tablet_manager.cc:1929] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:28.350142 18224 consensus_queue.cc:260] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.366523 18224 ts_tablet_manager.cc:1434] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
W20260924 19:57:28.366628 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8e27940040c495780f92f92431541cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:28.366940 18301 log.cc:1199] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/1b9ffd3fc8714979bfaa24c36d6fe0f9
I20260924 19:57:28.367769 18301 ts_tablet_manager.cc:1950] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:28.372110 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 30a3acc4b104447885b27ea75dca1bea (table default.Table_b013bc1fcd8b4e0db6c8d928b1984a3b [id=0fb7eef54311401c94d0c4e97de0cc0d]) successfully deleted
I20260924 19:57:28.372953 18220 tablet_bootstrap.cc:492] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:28.373345 18220 ts_tablet_manager.cc:1403] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.002s
I20260924 19:57:28.375747 18220 raft_consensus.cc:359] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.376344 18220 raft_consensus.cc:385] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.376575 18220 raft_consensus.cc:740] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:28.377239 18220 consensus_queue.cc:260] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.378396 18231 tablet_replica.cc:333] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:28.379081 18231 raft_consensus.cc:2243] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:28.383814 18231 pending_rounds.cc:70] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: Trying to abort 1 pending ops.
I20260924 19:57:28.384214 18231 pending_rounds.cc:77] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986252500525056 op_type: NO_OP noop_request { }
I20260924 19:57:28.391283 18231 raft_consensus.cc:2889] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:28.391767 18231 raft_consensus.cc:2272] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:28.403870 18220 ts_tablet_manager.cc:1434] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20260924 19:57:28.405858 18231 ts_tablet_manager.cc:1916] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:28.413909 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 5f994e0401df4fc19bc5abb10f08bc6e (DEFAULT_TABLE table=default.Table_e10b7d83fa124717bf2c0f53a065500b [id=b4e755ee98cd40388e3681825580d7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.414994 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f994e0401df4fc19bc5abb10f08bc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.416607 18220 tablet_bootstrap.cc:492] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:28.430819 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet fb011ae3f182464ab68d77924ab70181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:27 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:28.394662 18224 tablet_bootstrap.cc:492] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:28.436458 18224 tablet_bootstrap.cc:654] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:28.439981 17799 tablet_service.cc:1511] Processing CreateTablet for tablet 5f994e0401df4fc19bc5abb10f08bc6e (DEFAULT_TABLE table=default.Table_e10b7d83fa124717bf2c0f53a065500b [id=b4e755ee98cd40388e3681825580d7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.440903 18220 tablet_bootstrap.cc:654] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:28.441241 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f994e0401df4fc19bc5abb10f08bc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.447974 17615 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_14bb83c1f8ee444d81a998ed429a7b60" } modify_external_catalogs: true
I20260924 19:57:28.448566 17615 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_14bb83c1f8ee444d81a998ed429a7b60" } modify_external_catalogs: true
I20260924 19:57:28.467013 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_f80e5f6313d9425f9721192ea215cb74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:28.468755 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f80e5f6313d9425f9721192ea215cb74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:28.471184 18361 consensus_queue.cc:1048] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:28.472364 18223 tablet_bootstrap.cc:492] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:28.478072 18220 tablet_bootstrap.cc:492] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:28.478150 18223 tablet_bootstrap.cc:654] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:28.478494 18220 ts_tablet_manager.cc:1403] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.062s user 0.000s sys 0.009s
I20260924 19:57:28.480399 18224 tablet_bootstrap.cc:492] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:28.480813 18224 ts_tablet_manager.cc:1403] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.001s
I20260924 19:57:28.481468 18220 raft_consensus.cc:359] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.482232 18220 raft_consensus.cc:385] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.482481 18220 raft_consensus.cc:740] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:28.483319 18224 raft_consensus.cc:359] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.484007 18224 raft_consensus.cc:385] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.484256 18224 raft_consensus.cc:740] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:28.484908 18224 consensus_queue.cc:260] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.491909 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 (table default.Table_96afe0911af349bf9301ce80ef227bd0 [id=8edea4510fd84d40819b48b401ccfe64]) successfully deleted
I20260924 19:57:28.492861 18231 ts_tablet_manager.cc:1929] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:28.493196 18231 log.cc:1199] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/f33d435d7a594344956a88506a43ef59
I20260924 19:57:28.493904 18231 ts_tablet_manager.cc:1950] T f33d435d7a594344956a88506a43ef59 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:28.495221 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet f33d435d7a594344956a88506a43ef59 (table default.Table_b200a0c5e5014cc09c18338e29eb3e2a [id=fdb27eff83df4ca18ac25a6293459356]) successfully deleted
I20260924 19:57:28.495558 17951 tablet_service.cc:1511] Processing CreateTablet for tablet 5f994e0401df4fc19bc5abb10f08bc6e (DEFAULT_TABLE table=default.Table_e10b7d83fa124717bf2c0f53a065500b [id=b4e755ee98cd40388e3681825580d7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.496654 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f994e0401df4fc19bc5abb10f08bc6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.499430 18220 consensus_queue.cc:260] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.500581 18224 ts_tablet_manager.cc:1434] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260924 19:57:28.501976 18224 tablet_bootstrap.cc:492] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:28.504140 18292 raft_consensus.cc:493] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:28.504585 18292 raft_consensus.cc:515] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.506614 18292 leader_election.cc:290] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:28.509645 18220 ts_tablet_manager.cc:1434] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.031s user 0.000s sys 0.006s
I20260924 19:57:28.509733 18224 tablet_bootstrap.cc:654] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:28.513320 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90a677e94034c73ab5452554862c1c3" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:28.514083 18118 raft_consensus.cc:2468] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:28.515511 17905 leader_election.cc:304] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:28.517189 18292 raft_consensus.cc:2804] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:28.517530 18292 raft_consensus.cc:493] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:28.517890 18292 raft_consensus.cc:3060] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:28.523433 18361 raft_consensus.cc:493] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:28.523917 18361 raft_consensus.cc:515] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.527235 18361 leader_election.cc:290] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:28.529536 18224 tablet_bootstrap.cc:492] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:28.529300 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a384c9a0bb65429ebb97fe0807c78773" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:28.530040 18224 ts_tablet_manager.cc:1403] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.001s
W20260924 19:57:28.531894 18050 leader_election.cc:343] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:28.530735 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a384c9a0bb65429ebb97fe0807c78773" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
W20260924 19:57:28.533943 18051 leader_election.cc:343] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:28.534293 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 90cd09aecb6247e7b7937996f60f007f because the tablet was not found. No further retry: Not found: Tablet not found: 90cd09aecb6247e7b7937996f60f007f
I20260924 19:57:28.534341 18051 leader_election.cc:304] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:28.535926 18361 raft_consensus.cc:2749] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:28.533216 18224 raft_consensus.cc:359] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.536375 18224 raft_consensus.cc:385] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.536727 18224 raft_consensus.cc:740] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:28.537493 18224 consensus_queue.cc:260] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:28.538565 18220 tablet_bootstrap.cc:492] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:28.524945 18292 raft_consensus.cc:515] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.541359 18224 ts_tablet_manager.cc:1434] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.011s user 0.004s sys 0.003s
I20260924 19:57:28.541812 18223 tablet_bootstrap.cc:492] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:28.542169 18223 ts_tablet_manager.cc:1403] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.070s user 0.018s sys 0.015s
I20260924 19:57:28.542217 18224 tablet_bootstrap.cc:492] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:28.544811 18231 tablet_replica.cc:333] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:28.545514 18231 raft_consensus.cc:2243] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:28.546141 18231 pending_rounds.cc:70] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: Trying to abort 1 pending ops.
I20260924 19:57:28.546357 18231 pending_rounds.cc:77] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986256996315136 op_type: NO_OP noop_request { }
I20260924 19:57:28.546653 18231 raft_consensus.cc:2889] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:28.548367 18348 ts_tablet_manager.cc:1929] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:28.548753 18348 log.cc:1199] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/30a3acc4b104447885b27ea75dca1bea
I20260924 19:57:28.549508 18348 ts_tablet_manager.cc:1950] T 30a3acc4b104447885b27ea75dca1bea P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:28.551828 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 34) took 1389 ms. Trace:
I20260924 19:57:28.552173 18348 rpcz_store.cc:276] 0924 19:57:27.162417 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:27.168213 (+ 5796us) service_pool.cc:224] Handling call
0924 19:57:28.054657 (+886444us) ts_tablet_manager.cc:1151] Deleting tablet 30a3acc4b104447885b27ea75dca1bea
0924 19:57:28.054706 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:28.295442 (+240736us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:28.548220 (+252778us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:28.551753 (+ 3533us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":885916}
I20260924 19:57:28.554291 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 36) took 1305 ms. Trace:
I20260924 19:57:28.554606 18348 rpcz_store.cc:276] 0924 19:57:27.248621 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:27.248743 (+ 122us) service_pool.cc:224] Handling call
0924 19:57:28.553997 (+1305254us) ts_tablet_manager.cc:1151] Deleting tablet 6edb7999719e4c3c93b93539bfe1550c
0924 19:57:28.554041 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:28.554230 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1304722}
I20260924 19:57:28.555677 18348 tablet_replica.cc:333] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:28.556437 18348 raft_consensus.cc:2243] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:28.556903 18348 raft_consensus.cc:2272] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:28.559137 17602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20260924 19:57:28.564807 18348 ts_tablet_manager.cc:1916] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:28.570498 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 97a1b8ccd35a43cea7a71d705fad1494 (DEFAULT_TABLE table=default.Table_9a7c568838e54aa8b0ea15eeb70d7cd0 [id=f768ce157a7b407d9f1e55a5b9d0720a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.571624 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97a1b8ccd35a43cea7a71d705fad1494. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.573841 18220 tablet_bootstrap.cc:654] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:28.573562 18223 raft_consensus.cc:359] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.574200 18223 raft_consensus.cc:385] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.574452 18223 raft_consensus.cc:740] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:28.576776 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90a677e94034c73ab5452554862c1c3" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:28.577307 18118 raft_consensus.cc:3060] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:28.579741 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 30a3acc4b104447885b27ea75dca1bea (table default.Table_b013bc1fcd8b4e0db6c8d928b1984a3b [id=0fb7eef54311401c94d0c4e97de0cc0d]) successfully deleted
I20260924 19:57:28.546972 18231 raft_consensus.cc:2272] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:28.548106 18224 tablet_bootstrap.cc:654] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:28.580468 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 6edb7999719e4c3c93b93539bfe1550c because the tablet was not found. No further retry: Not found: Tablet not found: 6edb7999719e4c3c93b93539bfe1550c
I20260924 19:57:28.580376 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90a677e94034c73ab5452554862c1c3" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:28.571586 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 97a1b8ccd35a43cea7a71d705fad1494 (DEFAULT_TABLE table=default.Table_9a7c568838e54aa8b0ea15eeb70d7cd0 [id=f768ce157a7b407d9f1e55a5b9d0720a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.581002 17949 tablet_service.cc:1511] Processing CreateTablet for tablet 97a1b8ccd35a43cea7a71d705fad1494 (DEFAULT_TABLE table=default.Table_9a7c568838e54aa8b0ea15eeb70d7cd0 [id=f768ce157a7b407d9f1e55a5b9d0720a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.581954 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90a677e94034c73ab5452554862c1c3" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:28.581959 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97a1b8ccd35a43cea7a71d705fad1494. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.585079 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97a1b8ccd35a43cea7a71d705fad1494. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.590795 18118 raft_consensus.cc:2468] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
W20260924 19:57:28.592178 17906 leader_election.cc:343] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:28.593722 17906 leader_election.cc:343] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:28.597841 17798 tablet_service.cc:1511] Processing CreateTablet for tablet dd2f548d5bd5454cb16335473a9eeb03 (DEFAULT_TABLE table=default.Table_9740734f1493496d94dac20cb0e65c85 [id=0ceede10f5ea46729576babbd0df0c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.598979 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2f548d5bd5454cb16335473a9eeb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.598163 18095 tablet_service.cc:1511] Processing CreateTablet for tablet dd2f548d5bd5454cb16335473a9eeb03 (DEFAULT_TABLE table=default.Table_9740734f1493496d94dac20cb0e65c85 [id=0ceede10f5ea46729576babbd0df0c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.575073 18223 consensus_queue.cc:260] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.600948 17802 tablet_service.cc:1511] Processing CreateTablet for tablet efd7339d0fee436299a7eac02a82addd (DEFAULT_TABLE table=default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.607105 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2f548d5bd5454cb16335473a9eeb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.607805 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd7339d0fee436299a7eac02a82addd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.607915 18223 ts_tablet_manager.cc:1434] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.066s user 0.006s sys 0.000s
I20260924 19:57:28.622575 17799 tablet_service.cc:1511] Processing CreateTablet for tablet f482b53a4a2d44f6b3431c588cd605ce (DEFAULT_TABLE table=default.Table_b814c73599bb4d9fae83012912129997 [id=abad09090ec04805bf6260c67fd479c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.628374 17607 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 8a19b5812dc54589a5cbf48199951600
I20260924 19:57:28.630270 17607 catalog_manager.cc:6934] ReplaceTablet: tablet 8a19b5812dc54589a5cbf48199951600 deleted and replaced by tablet f38188eccc344246b72a8755ec5201ed
I20260924 19:57:28.633826 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f482b53a4a2d44f6b3431c588cd605ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.635347 18096 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a19b5812dc54589a5cbf48199951600 with delete_type TABLET_DATA_DELETED (replaced by tablet f38188eccc344246b72a8755ec5201ed) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:28.635695 17612 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet efd7339d0fee436299a7eac02a82addd
I20260924 19:57:28.636139 18301 tablet_replica.cc:333] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:28.636871 18301 raft_consensus.cc:2243] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:28.637300 18301 raft_consensus.cc:2272] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:28.638798 18096 tablet_service.cc:1558] Processing DeleteTablet for tablet efd7339d0fee436299a7eac02a82addd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:28.638984 18301 ts_tablet_manager.cc:1916] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:28.639810 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a19b5812dc54589a5cbf48199951600 with delete_type TABLET_DATA_DELETED (replaced by tablet f38188eccc344246b72a8755ec5201ed) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:28.631011 17607 master_service.cc:960] ReplaceTablet: received request to replace tablet f2897faafbff460bae99990a5940d38b from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:28.640813 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet a384c9a0bb65429ebb97fe0807c78773 (table default.Table_1a379ab4fc834d69865de7106e463200 [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 1 got 0
I20260924 19:57:28.641541 18097 tablet_service.cc:1511] Processing CreateTablet for tablet efd7339d0fee436299a7eac02a82addd (DEFAULT_TABLE table=default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.642575 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd7339d0fee436299a7eac02a82addd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.641960 17610 catalog_manager.cc:5838] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:28.644172 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 2 got 0
I20260924 19:57:28.645604 17610 catalog_manager.cc:5838] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:28.647452 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_8418d6fb70a647f896eb720d1958947c [id=31e1c3788d264823a12ecd9755ec2027]). Expected version 3 got 0
I20260924 19:57:28.648096 18362 consensus_queue.cc:1048] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:28.652709 18223 tablet_bootstrap.cc:492] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:28.658298 18223 tablet_bootstrap.cc:654] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:28.616652 17905 leader_election.cc:304] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
W20260924 19:57:28.659181 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 65453.1us
I20260924 19:57:28.659979 18403 raft_consensus.cc:2804] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:28.619318 18292 leader_election.cc:290] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:28.661268 17951 tablet_service.cc:1511] Processing CreateTablet for tablet dd2f548d5bd5454cb16335473a9eeb03 (DEFAULT_TABLE table=default.Table_9740734f1493496d94dac20cb0e65c85 [id=0ceede10f5ea46729576babbd0df0c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.662410 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2f548d5bd5454cb16335473a9eeb03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.666822 18096 tablet_service.cc:1511] Processing CreateTablet for tablet f482b53a4a2d44f6b3431c588cd605ce (DEFAULT_TABLE table=default.Table_b814c73599bb4d9fae83012912129997 [id=abad09090ec04805bf6260c67fd479c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.667923 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f482b53a4a2d44f6b3431c588cd605ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.670804 17797 tablet_service.cc:1558] Processing DeleteTablet for tablet efd7339d0fee436299a7eac02a82addd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:28.677471 18301 ts_tablet_manager.cc:1929] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:28.694096 18403 raft_consensus.cc:697] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:28.694012 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_bba088bd13274e99b23dfe9f5dd6c14a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:28.695087 18403 consensus_queue.cc:237] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
W20260924 19:57:28.696092 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bba088bd13274e99b23dfe9f5dd6c14a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:28.701321 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_fa88730066c84caa82e692ad0e8bb34c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:28.702858 18231 ts_tablet_manager.cc:1916] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:28.703706 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa88730066c84caa82e692ad0e8bb34c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:28.706411 17950 tablet_service.cc:1511] Processing CreateTablet for tablet efd7339d0fee436299a7eac02a82addd (DEFAULT_TABLE table=default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.707309 18220 tablet_bootstrap.cc:492] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:28.707530 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd7339d0fee436299a7eac02a82addd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.707706 18220 ts_tablet_manager.cc:1403] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.171s user 0.009s sys 0.002s
I20260924 19:57:28.710041 18220 raft_consensus.cc:359] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.710685 18220 raft_consensus.cc:385] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:57:28.718858 17613 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83ee43b6dc064dc7b320691d52e57ea7 consensus state has no leader
I20260924 19:57:28.710932 18220 raft_consensus.cc:740] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:28.724537 18224 tablet_bootstrap.cc:492] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:28.724988 18224 ts_tablet_manager.cc:1403] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.183s user 0.013s sys 0.004s
I20260924 19:57:28.726038 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 7c7b8422753a4746bde71fcb25409949 (DEFAULT_TABLE table=default.Table_388c160758d44e1e89b8cb5d541316f8 [id=70f30a75b8d443a0ae28243adf40862a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.728909 17615 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 1ad0ddbb36234bf99e5c1e73fe82e739
I20260924 19:57:28.728965 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7b8422753a4746bde71fcb25409949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.743889 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet efd7339d0fee436299a7eac02a82addd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:28.743822 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a19b5812dc54589a5cbf48199951600 with delete_type TABLET_DATA_DELETED (replaced by tablet f38188eccc344246b72a8755ec5201ed) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:28.746052 18224 raft_consensus.cc:359] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.746742 18224 raft_consensus.cc:385] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.751852 17947 tablet_service.cc:1511] Processing CreateTablet for tablet f482b53a4a2d44f6b3431c588cd605ce (DEFAULT_TABLE table=default.Table_b814c73599bb4d9fae83012912129997 [id=abad09090ec04805bf6260c67fd479c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.752936 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f482b53a4a2d44f6b3431c588cd605ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.755424 18220 consensus_queue.cc:260] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.760202 17952 tablet_service.cc:1511] Processing CreateTablet for tablet 7c7b8422753a4746bde71fcb25409949 (DEFAULT_TABLE table=default.Table_388c160758d44e1e89b8cb5d541316f8 [id=70f30a75b8d443a0ae28243adf40862a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.761186 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7b8422753a4746bde71fcb25409949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.747004 18224 raft_consensus.cc:740] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:28.764268 18220 ts_tablet_manager.cc:1434] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.056s user 0.007s sys 0.004s
I20260924 19:57:28.763846 18224 consensus_queue.cc:260] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.771698 18220 tablet_bootstrap.cc:492] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:28.721369 18095 tablet_service.cc:1511] Processing CreateTablet for tablet 7c7b8422753a4746bde71fcb25409949 (DEFAULT_TABLE table=default.Table_388c160758d44e1e89b8cb5d541316f8 [id=70f30a75b8d443a0ae28243adf40862a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:28.789198 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c7b8422753a4746bde71fcb25409949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:28.766603 18223 tablet_bootstrap.cc:492] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:28.803553 18223 ts_tablet_manager.cc:1403] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.151s user 0.010s sys 0.000s
I20260924 19:57:28.813881 18231 ts_tablet_manager.cc:1929] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:28.815188 18231 log.cc:1199] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/f1d456a7ccbe4f03b41d6b52ae382424
I20260924 19:57:28.815460 18224 ts_tablet_manager.cc:1434] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.090s user 0.006s sys 0.001s
I20260924 19:57:28.816025 18231 ts_tablet_manager.cc:1950] T f1d456a7ccbe4f03b41d6b52ae382424 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:28.804003 18301 log.cc:1199] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/8a19b5812dc54589a5cbf48199951600
I20260924 19:57:28.819506 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_3bf03540f35b468dacc7a86dd2d3d2a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:28.819945 18301 ts_tablet_manager.cc:1950] T 8a19b5812dc54589a5cbf48199951600 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:28.820465 18220 tablet_bootstrap.cc:654] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:28.821229 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bf03540f35b468dacc7a86dd2d3d2a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:28.821439 18231 tablet_replica.cc:333] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:28.821698 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet f1d456a7ccbe4f03b41d6b52ae382424 (table default.Table_4a15e8e61f934d08b802fd02d421cd3f [id=ef215e1163d74af69a3a3f6f763c3249]) successfully deleted
I20260924 19:57:28.822115 18231 raft_consensus.cc:2243] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:28.823014 18231 raft_consensus.cc:2272] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:28.823659 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 8a19b5812dc54589a5cbf48199951600 (table default.Table_37482fb26ba144619bb66861dceaab3b [id=06b8b073d6ae475c8125613b89463d3c]) successfully deleted
I20260924 19:57:28.825206 18231 ts_tablet_manager.cc:1916] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:28.825441 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet efd7339d0fee436299a7eac02a82addd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efd7339d0fee436299a7eac02a82addd already in progress: creating tablet
I20260924 19:57:28.831490 18411 raft_consensus.cc:493] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:28.831936 18411 raft_consensus.cc:515] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.833954 18411 leader_election.cc:290] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:28.835264 18348 ts_tablet_manager.cc:1929] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:28.836257 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b04059219084f608e8cb5fa4d0e0d58" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:28.836242 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b04059219084f608e8cb5fa4d0e0d58" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:28.835744 18348 log.cc:1199] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/a9505fd426134d068b95639012112bf3
W20260924 19:57:28.837809 18051 leader_election.cc:343] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:28.838321 18050 leader_election.cc:343] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:28.839003 18224 tablet_bootstrap.cc:492] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:28.838737 18050 leader_election.cc:304] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:28.840487 18411 raft_consensus.cc:2749] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:28.840869 18348 ts_tablet_manager.cc:1950] T a9505fd426134d068b95639012112bf3 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:28.842924 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 37) took 1583 ms. Trace:
I20260924 19:57:28.843371 18348 rpcz_store.cc:276] 0924 19:57:27.259304 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:27.273943 (+ 14639us) service_pool.cc:224] Handling call
0924 19:57:28.555525 (+1281582us) ts_tablet_manager.cc:1151] Deleting tablet a9505fd426134d068b95639012112bf3
0924 19:57:28.555568 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:28.779368 (+223800us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:28.835109 (+ 55741us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:28.842871 (+ 7762us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1281010}
I20260924 19:57:28.845242 18348 tablet_replica.cc:333] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:28.846127 18348 raft_consensus.cc:2243] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:28.852924 17615 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_c2dba817ef2249bd8c401ea137c347e7" } modify_external_catalogs: true
I20260924 19:57:28.853497 17615 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_c2dba817ef2249bd8c401ea137c347e7" } modify_external_catalogs: true
I20260924 19:57:28.855001 18348 raft_consensus.cc:2272] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:28.857188 18348 ts_tablet_manager.cc:1916] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:28.857730 18220 tablet_bootstrap.cc:492] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:28.858114 18220 ts_tablet_manager.cc:1403] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.000s
I20260924 19:57:28.860389 18220 raft_consensus.cc:359] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.861002 18220 raft_consensus.cc:385] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.861258 18220 raft_consensus.cc:740] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:28.871452 18220 consensus_queue.cc:260] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.875126 18231 ts_tablet_manager.cc:1929] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:28.875612 18231 log.cc:1199] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/f2897faafbff460bae99990a5940d38b
I20260924 19:57:28.876430 18231 ts_tablet_manager.cc:1950] T f2897faafbff460bae99990a5940d38b P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:28.878124 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet f2897faafbff460bae99990a5940d38b (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]) successfully deleted
I20260924 19:57:28.879261 18231 tablet_replica.cc:333] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:28.881273 18231 raft_consensus.cc:2243] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:28.881773 18231 raft_consensus.cc:2272] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:28.847915 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet a9505fd426134d068b95639012112bf3 (table default.Table_9740734f1493496d94dac20cb0e65c85 [id=0ceede10f5ea46729576babbd0df0c56]) successfully deleted
I20260924 19:57:28.890901 18403 raft_consensus.cc:493] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:28.891139 17609 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 954178b8cc1e434085337f66b0e85672 has no consensus state
I20260924 19:57:28.806751 18223 raft_consensus.cc:359] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.891547 18223 raft_consensus.cc:385] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.891400 18403 raft_consensus.cc:515] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.891723 17609 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 87) took 1160 ms. Trace:
I20260924 19:57:28.891850 18223 raft_consensus.cc:740] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
W20260924 19:57:28.891847 17614 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 28) took 948 ms (client timeout 1000 ms). Trace:
W20260924 19:57:28.892220 17614 rpcz_store.cc:269] 0924 19:57:27.943615 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:27.943893 (+ 278us) service_pool.cc:224] Handling call
0924 19:57:28.891762 (+947869us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260924 19:57:28.892563 18223 consensus_queue.cc:260] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.893388 18403 leader_election.cc:290] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:28.894884 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a384c9a0bb65429ebb97fe0807c78773" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:28.895076 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a384c9a0bb65429ebb97fe0807c78773" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:28.895506 18118 raft_consensus.cc:2468] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
W20260924 19:57:28.896672 17906 leader_election.cc:343] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:28.899076 18231 ts_tablet_manager.cc:1916] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:28.899358 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet f2897faafbff460bae99990a5940d38b with delete_type TABLET_DATA_DELETED (replaced by tablet 8a524bad1c4040e58ad84fcf5a4fc47d) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:28.900269 17905 leader_election.cc:304] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
W20260924 19:57:28.901147 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet f2897faafbff460bae99990a5940d38b because the tablet was not found. No further retry: Not found: Tablet not found: f2897faafbff460bae99990a5940d38b
I20260924 19:57:28.901413 18403 raft_consensus.cc:2804] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:28.901600 18096 tablet_service.cc:1558] Processing DeleteTablet for tablet 90cd09aecb6247e7b7937996f60f007f with delete_type TABLET_DATA_DELETED (replaced by tablet efd7339d0fee436299a7eac02a82addd) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:28.901808 18403 raft_consensus.cc:493] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:28.902390 18403 raft_consensus.cc:3060] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:28.906015 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 90cd09aecb6247e7b7937996f60f007f because the tablet was not found. No further retry: Not found: Tablet not found: 90cd09aecb6247e7b7937996f60f007f
W20260924 19:57:28.898876 17608 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 17) took 1704 ms (client timeout 1000 ms). Trace:
W20260924 19:57:28.907927 17608 rpcz_store.cc:269] 0924 19:57:27.193984 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:27.199452 (+ 5468us) service_pool.cc:224] Handling call
0924 19:57:28.898802 (+1699350us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":4}
I20260924 19:57:28.910236 18348 ts_tablet_manager.cc:1929] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:28.914894 17823 raft_consensus.cc:1275] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Refusing update from remote peer eaf64bfb64aa42948c9bc1411ce4ce35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:57:28.916751 18362 consensus_queue.cc:1048] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:28.892058 17609 rpcz_store.cc:276] 0924 19:57:27.731628 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:27.792344 (+ 60716us) service_pool.cc:224] Handling call
0924 19:57:27.793571 (+ 1227us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:27.794442 (+ 871us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:27.795781 (+ 1339us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:27.796025 (+ 244us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:27.998867 (+202842us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:28.891639 (+892772us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:27.945152 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:27.945561 (+ 409us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:27.945585 (+ 24us) write_op.cc:435] Acquired schema lock
0924 19:57:27.945602 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:27.945858 (+ 256us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:27.945897 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:27.945923 (+ 26us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:27.946071 (+ 148us) tablet.cc:701] Row locks acquired
0924 19:57:27.946085 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:27.946191 (+ 106us) write_op.cc:270] Start()
0924 19:57:27.946279 (+ 88us) write_op.cc:276] Timestamp: P: 1790279847946140 usec, L: 0
0924 19:57:27.946296 (+ 17us) op_driver.cc:348] REPLICATION: starting
0924 19:57:27.946752 (+ 456us) log.cc:844] Serialized 404 byte log entry
0924 19:57:27.996762 (+ 50010us) op_driver.cc:464] REPLICATION: finished
0924 19:57:27.997413 (+ 651us) write_op.cc:301] APPLY: starting
0924 19:57:27.997472 (+ 59us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:27.997568 (+ 96us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:27.997705 (+ 137us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:27.997856 (+ 151us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:27.997998 (+ 142us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:27.998047 (+ 49us) write_op.cc:312] APPLY: finished
0924 19:57:27.998256 (+ 209us) log.cc:844] Serialized 22 byte log entry
0924 19:57:27.998411 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:27.998471 (+ 60us) write_op.cc:454] Released schema lock
0924 19:57:27.998503 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:27.999259 (+ 756us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:27.999288 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":478,"apply.run_cpu_time_us":1334,"apply.run_wall_time_us":1787,"num_ops":1,"prepare.queue_time_us":147670,"prepare.run_cpu_time_us":1862,"prepare.run_wall_time_us":1861,"raft.queue_time_us":878,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":50368}]]}
I20260924 19:57:28.919401 18223 ts_tablet_manager.cc:1434] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.115s user 0.002s sys 0.004s
I20260924 19:57:28.927013 17611 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_e10b7d83fa124717bf2c0f53a065500b" } new_table_name: "default.Table_b2cb17472c5b450c93e06db475f000a0" modify_external_catalogs: true
I20260924 19:57:28.940829 18224 tablet_bootstrap.cc:654] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:28.942112 18223 tablet_bootstrap.cc:492] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:28.952443 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet a9505fd426134d068b95639012112bf3 with delete_type TABLET_DATA_DELETED (replaced by tablet dd2f548d5bd5454cb16335473a9eeb03) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:28.953135 18096 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a19b5812dc54589a5cbf48199951600 with delete_type TABLET_DATA_DELETED (replaced by tablet f38188eccc344246b72a8755ec5201ed) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:28.953826 18096 tablet_service.cc:1558] Processing DeleteTablet for tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 954178b8cc1e434085337f66b0e85672) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:28.955037 17600 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 954178b8cc1e434085337f66b0e85672 has no consensus state
I20260924 19:57:28.959419 18301 tablet_replica.cc:333] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:28.960199 18301 raft_consensus.cc:2243] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:28.960690 18301 raft_consensus.cc:2272] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:28.962383 18301 ts_tablet_manager.cc:1916] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:28.962898 17607 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_850acd445f004facb3a20aaac28003e3" } modify_external_catalogs: true
I20260924 19:57:28.963488 17607 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_850acd445f004facb3a20aaac28003e3" } modify_external_catalogs: true
I20260924 19:57:28.965483 17615 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 3621bc6e14514145b9f1f3ccdbae3db0
I20260924 19:57:28.968866 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_e3179b11ea114ce99779ed0b2ac46732"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:28.971083 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3179b11ea114ce99779ed0b2ac46732 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:28.971901 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_06a8017f949541c0bbe0df4972ebc15d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:28.974057 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06a8017f949541c0bbe0df4972ebc15d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:28.981986 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 8a19b5812dc54589a5cbf48199951600 because the tablet was not found. No further retry: Not found: Tablet not found: 8a19b5812dc54589a5cbf48199951600
I20260924 19:57:28.984117 18224 tablet_bootstrap.cc:492] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
W20260924 19:57:28.984015 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 because the tablet was not found. No further retry: Not found: Tablet not found: 1b9ffd3fc8714979bfaa24c36d6fe0f9
I20260924 19:57:28.984521 18224 ts_tablet_manager.cc:1403] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.149s user 0.009s sys 0.000s
I20260924 19:57:28.986892 18224 raft_consensus.cc:359] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.992136 18224 raft_consensus.cc:385] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:28.992408 18224 raft_consensus.cc:740] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:28.998209 18223 tablet_bootstrap.cc:654] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:28.998718 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 2 got 0
I20260924 19:57:28.921468 17970 raft_consensus.cc:1275] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Refusing update from remote peer eaf64bfb64aa42948c9bc1411ce4ce35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:57:28.999675 18292 raft_consensus.cc:493] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:29.000123 18292 raft_consensus.cc:515] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.000871 18362 consensus_queue.cc:1048] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:29.001137 18348 log.cc:1199] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/fb011ae3f182464ab68d77924ab70181
I20260924 19:57:28.941382 18220 ts_tablet_manager.cc:1434] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.083s user 0.007s sys 0.001s
I20260924 19:57:28.911083 18403 raft_consensus.cc:515] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:28.958392 18413 raft_consensus.cc:493] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:29.007603 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81f74c7ba2443e49228b17e867588cc" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:29.008381 18118 raft_consensus.cc:2468] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:29.009630 18290 raft_consensus.cc:493] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:29.002167 18292 leader_election.cc:290] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:29.010174 18290 raft_consensus.cc:515] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:29.004055 18220 tablet_bootstrap.cc:492] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:29.012280 18290 leader_election.cc:290] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:29.013010 17822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a384c9a0bb65429ebb97fe0807c78773" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:29.005509 18403 leader_election.cc:290] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:29.016543 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81f74c7ba2443e49228b17e867588cc" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:29.016875 18220 tablet_bootstrap.cc:654] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:29.006474 18413 raft_consensus.cc:515] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.018442 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9505fd426134d068b95639012112bf3" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:29.019048 17906 leader_election.cc:343] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:29.019706 17906 leader_election.cc:343] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:29.020537 18413 leader_election.cc:290] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:29.020869 17753 leader_election.cc:343] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Not found: Tablet not found: a9505fd426134d068b95639012112bf3
I20260924 19:57:29.022550 17905 leader_election.cc:304] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
I20260924 19:57:29.024914 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b04059219084f608e8cb5fa4d0e0d58" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:29.025504 18117 raft_consensus.cc:2468] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:29.025426 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b04059219084f608e8cb5fa4d0e0d58" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:29.026144 18413 raft_consensus.cc:2804] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:29.026585 18413 raft_consensus.cc:493] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:29.027416 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a384c9a0bb65429ebb97fe0807c78773" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:29.028055 18118 raft_consensus.cc:3060] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:29.028096 17905 leader_election.cc:304] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:29.029138 18403 raft_consensus.cc:2804] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:29.029541 18403 raft_consensus.cc:493] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:29.029919 18403 raft_consensus.cc:3060] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:29.001905 18348 ts_tablet_manager.cc:1950] T fb011ae3f182464ab68d77924ab70181 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
W20260924 19:57:29.021037 17602 consensus_peers.cc:597] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e -> Peer 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309): Couldn't send request to peer 3e059bbe4dac4178bccb3be48cb1118d. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260924 19:57:29.034749 17753 leader_election.cc:343] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: Tablet not RUNNING: STOPPED
I20260924 19:57:29.035212 17753 leader_election.cc:304] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:29.036587 17906 leader_election.cc:343] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:29.037029 18403 raft_consensus.cc:515] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
W20260924 19:57:29.039382 17611 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f994e0401df4fc19bc5abb10f08bc6e consensus state has no leader
I20260924 19:57:29.040962 18290 raft_consensus.cc:2749] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:29.042735 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_bc465c5a40994196b5e0996ee2176340"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:29.044358 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc465c5a40994196b5e0996ee2176340 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:29.045996 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b04059219084f608e8cb5fa4d0e0d58" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:29.046577 18117 raft_consensus.cc:3060] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:29.046152 18403 leader_election.cc:290] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:29.048194 18118 raft_consensus.cc:2468] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:29.050002 17905 leader_election.cc:304] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
I20260924 19:57:29.051097 18403 raft_consensus.cc:2804] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:29.051587 18403 raft_consensus.cc:697] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:29.052359 18403 consensus_queue.cc:237] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.052870 18224 consensus_queue.cc:260] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.055030 17611 master_service.cc:960] ReplaceTablet: received request to replace tablet efd7339d0fee436299a7eac02a82addd from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:29.062376 18117 raft_consensus.cc:2468] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:29.068513 17905 leader_election.cc:304] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:29.069782 18426 raft_consensus.cc:2804] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:29.071887 18426 raft_consensus.cc:697] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:29.073422 18426 consensus_queue.cc:237] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.037387 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 40) took 1698 ms. Trace:
I20260924 19:57:29.077006 18220 tablet_bootstrap.cc:492] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:29.026955 18413 raft_consensus.cc:3060] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:29.077471 18220 ts_tablet_manager.cc:1403] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.001s
I20260924 19:57:29.076918 18348 rpcz_store.cc:276] 0924 19:57:27.339089 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:27.339251 (+ 162us) service_pool.cc:224] Handling call
0924 19:57:28.845048 (+1505797us) ts_tablet_manager.cc:1151] Deleting tablet fb011ae3f182464ab68d77924ab70181
0924 19:57:28.845092 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:28.868768 (+ 23676us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:28.910106 (+ 41338us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.037324 (+127218us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1473923}
I20260924 19:57:29.078378 18348 tablet_replica.cc:333] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:29.079174 18348 raft_consensus.cc:2243] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:29.079679 17612 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 2 got 0
I20260924 19:57:29.080446 18220 raft_consensus.cc:359] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.081152 18220 raft_consensus.cc:385] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:29.081406 18220 raft_consensus.cc:740] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:29.084230 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet a384c9a0bb65429ebb97fe0807c78773 (table default.Table_1a379ab4fc834d69865de7106e463200 [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 1 got 0
I20260924 19:57:29.085111 17609 catalog_manager.cc:5838] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:29.082540 18220 consensus_queue.cc:260] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.089514 18220 ts_tablet_manager.cc:1434] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260924 19:57:29.090432 18220 tablet_bootstrap.cc:492] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:29.084853 18348 raft_consensus.cc:2272] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:29.085062 18413 raft_consensus.cc:515] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.092882 18413 leader_election.cc:290] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:29.094972 18348 ts_tablet_manager.cc:1916] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:29.096042 18223 tablet_bootstrap.cc:492] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:29.096475 18223 ts_tablet_manager.cc:1403] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.176s user 0.004s sys 0.006s
I20260924 19:57:29.099555 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 2 got 0
I20260924 19:57:29.100525 17607 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 9b04059219084f608e8cb5fa4d0e0d58
I20260924 19:57:29.108827 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet fb011ae3f182464ab68d77924ab70181 (table default.Table_3d0e38cde76d4300a129a04a04b5915d [id=2d056489f9314bb59c83e506d47ca5b8]) successfully deleted
I20260924 19:57:29.110191 18348 ts_tablet_manager.cc:1929] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:29.110630 18348 log.cc:1199] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/f1d456a7ccbe4f03b41d6b52ae382424
I20260924 19:57:29.111064 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet 9b04059219084f608e8cb5fa4d0e0d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:29.111409 18348 ts_tablet_manager.cc:1950] T f1d456a7ccbe4f03b41d6b52ae382424 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:29.112378 18220 tablet_bootstrap.cc:654] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:29.113188 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 42) took 1239 ms. Trace:
I20260924 19:57:29.113497 18348 rpcz_store.cc:276] 0924 19:57:27.873400 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:27.876550 (+ 3150us) service_pool.cc:224] Handling call
0924 19:57:29.078204 (+1201654us) ts_tablet_manager.cc:1151] Deleting tablet f1d456a7ccbe4f03b41d6b52ae382424
0924 19:57:29.078254 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:29.102736 (+ 24482us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.110069 (+ 7333us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.113128 (+ 3059us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5123,"tablet-delete.queue_time_us":1184873}
I20260924 19:57:29.114985 18348 tablet_replica.cc:333] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:29.115697 18348 raft_consensus.cc:2243] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:29.116214 18348 raft_consensus.cc:2272] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:29.118850 18348 ts_tablet_manager.cc:1916] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:29.098862 18223 raft_consensus.cc:359] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.132382 18223 raft_consensus.cc:385] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:29.132709 18223 raft_consensus.cc:740] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:29.139524 18223 consensus_queue.cc:260] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.158596 18348 ts_tablet_manager.cc:1929] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:29.159921 18301 ts_tablet_manager.cc:1929] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:29.162487 18301 log.cc:1199] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/a9505fd426134d068b95639012112bf3
I20260924 19:57:29.163741 18301 ts_tablet_manager.cc:1950] T a9505fd426134d068b95639012112bf3 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:29.174973 17615 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_a4745efada6f4edea3f2d4bad1c3905e" } modify_external_catalogs: true
I20260924 19:57:29.175634 17615 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_a4745efada6f4edea3f2d4bad1c3905e" } modify_external_catalogs: true
I20260924 19:57:29.177779 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_0bb01b569e3c41a4a53d6eb7817aaade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:29.180478 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bb01b569e3c41a4a53d6eb7817aaade in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:29.184581 18224 ts_tablet_manager.cc:1434] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.200s user 0.006s sys 0.000s
I20260924 19:57:29.185434 18224 tablet_bootstrap.cc:492] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:29.190515 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet a9505fd426134d068b95639012112bf3 (table default.Table_9740734f1493496d94dac20cb0e65c85 [id=0ceede10f5ea46729576babbd0df0c56]) successfully deleted
I20260924 19:57:29.160996 18223 ts_tablet_manager.cc:1434] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.064s user 0.007s sys 0.000s
I20260924 19:57:29.176581 18362 raft_consensus.cc:493] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:29.191371 18362 raft_consensus.cc:515] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.193378 18362 leader_election.cc:290] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:29.198028 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet 9b04059219084f608e8cb5fa4d0e0d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:29.199299 18290 raft_consensus.cc:493] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:29.199823 18290 raft_consensus.cc:515] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:29.198046 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet a9505fd426134d068b95639012112bf3 with delete_type TABLET_DATA_DELETED (replaced by tablet dd2f548d5bd5454cb16335473a9eeb03) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:29.201696 17609 master_service.cc:960] ReplaceTablet: received request to replace tablet fb011ae3f182464ab68d77924ab70181 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:29.201969 18290 leader_election.cc:290] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:29.207005 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f994e0401df4fc19bc5abb10f08bc6e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:29.205389 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b04059219084f608e8cb5fa4d0e0d58" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
W20260924 19:57:29.211081 18051 leader_election.cc:343] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:29.211582 18223 tablet_bootstrap.cc:492] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:29.214639 17753 leader_election.cc:343] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: fb011ae3f182464ab68d77924ab70181
I20260924 19:57:29.220310 18423 mvcc.cc:204] Tried to move back new op lower bound from 7332986261081427968 to 7332986256645443584. Current Snapshot: MvccSnapshot[applied={T|T < 7332986261081427968}]
I20260924 19:57:29.226382 18290 raft_consensus.cc:493] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:29.218425 18301 tablet_replica.cc:333] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:29.226949 18290 raft_consensus.cc:515] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.228008 18301 raft_consensus.cc:2243] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:29.229074 18290 leader_election.cc:290] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:29.235625 18301 raft_consensus.cc:2272] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:29.237462 18301 ts_tablet_manager.cc:1916] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:29.241075 17753 leader_election.cc:343] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 8a19b5812dc54589a5cbf48199951600
I20260924 19:57:29.244427 17609 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_bba088bd13274e99b23dfe9f5dd6c14a" } new_table_name: "default.Table_d944ee03376e4f54ae842e8c7dfb5b5c" modify_external_catalogs: true
I20260924 19:57:29.257279 18231 ts_tablet_manager.cc:1929] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:29.257805 18231 log.cc:1199] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/1b9ffd3fc8714979bfaa24c36d6fe0f9
I20260924 19:57:29.258548 18231 ts_tablet_manager.cc:1950] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:29.262775 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_1c71da8cadaa4593a1136522286f8d11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:29.247098 18224 tablet_bootstrap.cc:654] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:29.264623 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c71da8cadaa4593a1136522286f8d11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:29.266786 17610 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_9a7c568838e54aa8b0ea15eeb70d7cd0" } new_table_name: "default.Table_850e5158d78d4850b87445dac75ef7a9" modify_external_catalogs: true
I20260924 19:57:29.275171 18223 tablet_bootstrap.cc:654] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:29.284950 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 47) took 1163 ms. Trace:
W20260924 19:57:29.289860 17609 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c861e93ad9f40ba9ad010021e43bc2b has no consensus state
I20260924 19:57:29.292920 18224 tablet_bootstrap.cc:492] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:29.293370 18224 ts_tablet_manager.cc:1403] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.108s user 0.008s sys 0.001s
I20260924 19:57:29.293006 18434 tablet.cc:1751] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260924 19:57:29.295912 18224 raft_consensus.cc:359] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.296542 18224 raft_consensus.cc:385] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:29.296788 18224 raft_consensus.cc:740] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:29.305459 17608 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_f80e5f6313d9425f9721192ea215cb74" } modify_external_catalogs: true
I20260924 19:57:29.306099 17608 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_f80e5f6313d9425f9721192ea215cb74" } modify_external_catalogs: true
I20260924 19:57:29.307520 18224 consensus_queue.cc:260] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.309525 18224 ts_tablet_manager.cc:1434] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260924 19:57:29.310786 18224 tablet_bootstrap.cc:492] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:29.313112 18290 raft_consensus.cc:493] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:29.313715 18290 raft_consensus.cc:515] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.316071 18290 leader_election.cc:290] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:29.308233 18433 tablet.cc:1751] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260924 19:57:29.324747 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90a677e94034c73ab5452554862c1c3" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:29.325587 18117 raft_consensus.cc:2393] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c4f6e3a7b06148918d230c77ab0c8202 in current term 1: Already voted for candidate f74e6ed351284d7fb110e44336ce643a in this term.
I20260924 19:57:29.325630 18224 tablet_bootstrap.cc:654] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:29.285456 18231 rpcz_store.cc:276] 0924 19:57:28.121927 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:28.122045 (+ 118us) service_pool.cc:224] Handling call
0924 19:57:28.879079 (+757034us) ts_tablet_manager.cc:1151] Deleting tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9
0924 19:57:28.879126 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:29.116431 (+237305us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.257141 (+140710us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.284879 (+ 27738us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":756437}
I20260924 19:57:29.334249 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]) successfully deleted
W20260924 19:57:29.334901 17610 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97a1b8ccd35a43cea7a71d705fad1494 consensus state has no leader
I20260924 19:57:29.335786 18362 raft_consensus.cc:493] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:29.336228 18362 raft_consensus.cc:515] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.338428 18362 leader_election.cc:290] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:29.339145 18231 tablet_replica.cc:333] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:29.359586 17609 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_8be3db507b2d42da90969bacc361c127"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:29.361701 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81f74c7ba2443e49228b17e867588cc" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:29.362303 18117 raft_consensus.cc:3060] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:29.347043 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97a1b8ccd35a43cea7a71d705fad1494" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
W20260924 19:57:29.365113 18051 leader_election.cc:343] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:29.366379 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_11ca95d108694c9498d4de67c9ff8808"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:29.369855 18117 raft_consensus.cc:2468] T d81f74c7ba2443e49228b17e867588cc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:29.373580 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet f1d456a7ccbe4f03b41d6b52ae382424 (table default.Table_4a15e8e61f934d08b802fd02d421cd3f [id=ef215e1163d74af69a3a3f6f763c3249]) successfully deleted
I20260924 19:57:29.374750 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81f74c7ba2443e49228b17e867588cc" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
W20260924 19:57:29.379933 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11ca95d108694c9498d4de67c9ff8808 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:29.383061 17753 leader_election.cc:343] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Not found: Tablet not found: fb011ae3f182464ab68d77924ab70181
I20260924 19:57:29.383531 17753 leader_election.cc:304] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:29.384455 18435 raft_consensus.cc:2749] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:29.412523 18301 ts_tablet_manager.cc:1929] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:29.413882 18301 log.cc:1199] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/9b04059219084f608e8cb5fa4d0e0d58
W20260924 19:57:29.414229 17609 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8be3db507b2d42da90969bacc361c127 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:29.420215 18301 ts_tablet_manager.cc:1950] T 9b04059219084f608e8cb5fa4d0e0d58 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:29.425721 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 9b04059219084f608e8cb5fa4d0e0d58 (table default.Table_850acd445f004facb3a20aaac28003e3 [id=37ed98d304b9467fa4213115e9fd4da5]) successfully deleted
I20260924 19:57:29.430101 18231 raft_consensus.cc:2243] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:29.430586 18231 raft_consensus.cc:2272] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:29.433697 17610 master_service.cc:960] ReplaceTablet: received request to replace tablet a8cc802bf87f4d739175f93203478af8 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:29.438442 17611 master_service.cc:960] ReplaceTablet: received request to replace tablet dd2f548d5bd5454cb16335473a9eeb03 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:29.440577 18348 log.cc:1199] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/f2897faafbff460bae99990a5940d38b
I20260924 19:57:29.441674 18348 ts_tablet_manager.cc:1950] T f2897faafbff460bae99990a5940d38b P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:29.444640 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 43) took 1445 ms. Trace:
I20260924 19:57:29.444986 18348 rpcz_store.cc:276] 0924 19:57:27.999598 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:27.999855 (+ 257us) service_pool.cc:224] Handling call
0924 19:57:29.114848 (+1114993us) ts_tablet_manager.cc:1151] Deleting tablet f2897faafbff460bae99990a5940d38b
0924 19:57:29.114887 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:29.151451 (+ 36564us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.158472 (+ 7021us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.444547 (+286075us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":513,"tablet-delete.queue_time_us":1114394}
I20260924 19:57:29.371465 17905 leader_election.cc:304] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
W20260924 19:57:29.373847 17906 leader_election.cc:343] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:29.379343 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f90a677e94034c73ab5452554862c1c3" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:29.380882 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a19b5812dc54589a5cbf48199951600" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:29.385289 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97a1b8ccd35a43cea7a71d705fad1494" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:29.389181 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f994e0401df4fc19bc5abb10f08bc6e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:29.447942 18403 raft_consensus.cc:2804] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:29.449020 17967 raft_consensus.cc:2468] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:29.450306 18403 raft_consensus.cc:697] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:29.451447 18348 tablet_replica.cc:333] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:29.452289 18348 raft_consensus.cc:2243] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:29.452886 18348 raft_consensus.cc:2272] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:29.450111 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 77188.5us
I20260924 19:57:29.451471 18403 consensus_queue.cc:237] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
W20260924 19:57:29.456116 18050 leader_election.cc:343] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:29.456533 18050 leader_election.cc:304] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:29.458680 18362 raft_consensus.cc:2749] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:29.461004 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 87358.8us
I20260924 19:57:29.462257 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet 9b04059219084f608e8cb5fa4d0e0d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:29.463835 17906 leader_election.cc:343] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:29.463860 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet f2897faafbff460bae99990a5940d38b (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]) successfully deleted
I20260924 19:57:29.464779 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet fb011ae3f182464ab68d77924ab70181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:27 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:29.472415 18348 ts_tablet_manager.cc:1916] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:29.485039 18220 tablet_bootstrap.cc:492] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:29.485567 18220 ts_tablet_manager.cc:1403] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.395s user 0.008s sys 0.021s
I20260924 19:57:29.494864 17613 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_1c71da8cadaa4593a1136522286f8d11" } new_table_name: "default.Table_de89731e2fdb4f998ae007bd35b1a3f7" modify_external_catalogs: true
I20260924 19:57:29.502497 18095 tablet_service.cc:1511] Processing CreateTablet for tablet 916877c7540a4f6685064fc2807cb140 (DEFAULT_TABLE table=default.Table_f80e5f6313d9425f9721192ea215cb74 [id=a0bdd622a33241118711b9828d48e1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.503937 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 916877c7540a4f6685064fc2807cb140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.504504 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_a10031a0fa004ac3ad48e6020477494b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:29.506712 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a10031a0fa004ac3ad48e6020477494b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:29.511662 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_5f33677cfb6a4eab8497cae637ecdf26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:29.515527 18096 tablet_service.cc:1511] Processing CreateTablet for tablet 349c6e9413bb41268d0c41d549203e2a (DEFAULT_TABLE table=default.Table_2be07cf4cdd64ac5a99db4037ee35542 [id=b1d9f8b4039b4d19bfc493355e0d07cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.517045 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349c6e9413bb41268d0c41d549203e2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:29.523705 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f33677cfb6a4eab8497cae637ecdf26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:29.526985 18220 raft_consensus.cc:359] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.432312 18231 ts_tablet_manager.cc:1916] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:29.457044 18224 tablet_bootstrap.cc:492] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:29.509871 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 349c6e9413bb41268d0c41d549203e2a (DEFAULT_TABLE table=default.Table_2be07cf4cdd64ac5a99db4037ee35542 [id=b1d9f8b4039b4d19bfc493355e0d07cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.528645 18224 ts_tablet_manager.cc:1403] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.218s user 0.015s sys 0.025s
I20260924 19:57:29.488891 17753 leader_election.cc:304] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:29.529402 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349c6e9413bb41268d0c41d549203e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.529547 18435 raft_consensus.cc:3060] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:29.529987 17753 leader_election.cc:304] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters: eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:29.530726 18290 raft_consensus.cc:2804] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:29.531028 18290 raft_consensus.cc:493] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:29.501551 17799 tablet_service.cc:1511] Processing CreateTablet for tablet 916877c7540a4f6685064fc2807cb140 (DEFAULT_TABLE table=default.Table_f80e5f6313d9425f9721192ea215cb74 [id=a0bdd622a33241118711b9828d48e1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.531394 18290 raft_consensus.cc:3060] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:29.531114 18224 raft_consensus.cc:359] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.531735 18224 raft_consensus.cc:385] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:29.531982 18224 raft_consensus.cc:740] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:29.532059 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 916877c7540a4f6685064fc2807cb140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.532733 18224 consensus_queue.cc:260] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.549352 18223 tablet_bootstrap.cc:492] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:29.549786 18223 ts_tablet_manager.cc:1403] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.339s user 0.016s sys 0.016s
I20260924 19:57:29.558830 17947 tablet_service.cc:1511] Processing CreateTablet for tablet 916877c7540a4f6685064fc2807cb140 (DEFAULT_TABLE table=default.Table_f80e5f6313d9425f9721192ea215cb74 [id=a0bdd622a33241118711b9828d48e1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.560283 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 916877c7540a4f6685064fc2807cb140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.561897 18435 raft_consensus.cc:2749] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:29.527700 18220 raft_consensus.cc:385] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:29.565140 18220 raft_consensus.cc:740] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:29.565945 18433 tablet.cc:1746] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: Skipping requested alter to schema version 4, tablet already version 4
I20260924 19:57:29.546726 18290 raft_consensus.cc:515] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.572934 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a19b5812dc54589a5cbf48199951600" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:29.573172 18290 leader_election.cc:290] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:29.573490 17967 raft_consensus.cc:3060] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:29.587852 18220 consensus_queue.cc:260] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.593420 18224 ts_tablet_manager.cc:1434] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.064s user 0.008s sys 0.010s
I20260924 19:57:29.598461 18220 ts_tablet_manager.cc:1434] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.113s user 0.004s sys 0.004s
I20260924 19:57:29.600010 18434 tablet.cc:1746] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: Skipping requested alter to schema version 4, tablet already version 4
I20260924 19:57:29.600870 18220 tablet_bootstrap.cc:492] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:29.601876 18224 tablet_bootstrap.cc:492] T d81f74c7ba2443e49228b17e867588cc P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:29.602447 17949 tablet_service.cc:1511] Processing CreateTablet for tablet 349c6e9413bb41268d0c41d549203e2a (DEFAULT_TABLE table=default.Table_2be07cf4cdd64ac5a99db4037ee35542 [id=b1d9f8b4039b4d19bfc493355e0d07cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.603600 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349c6e9413bb41268d0c41d549203e2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:29.604900 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 148797us
I20260924 19:57:29.607853 18223 raft_consensus.cc:359] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:29.608566 18223 raft_consensus.cc:385] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:29.608814 18223 raft_consensus.cc:740] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:29.609452 18223 consensus_queue.cc:260] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:29.616853 17608 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 916877c7540a4f6685064fc2807cb140
I20260924 19:57:29.619169 17950 tablet_service.cc:1558] Processing DeleteTablet for tablet 916877c7540a4f6685064fc2807cb140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:29.621050 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 916877c7540a4f6685064fc2807cb140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:29.622666 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 916877c7540a4f6685064fc2807cb140 because the tablet was not found. No further retry: Not found: Tablet not found: 916877c7540a4f6685064fc2807cb140
I20260924 19:57:29.624825 17950 tablet_service.cc:1511] Processing CreateTablet for tablet ef3f82de70bb4328a15a044fe0e500bf (DEFAULT_TABLE table=default.Table_cca06980f109487a9ed8b8172950b60d [id=016f4039733144c985808b862eebcc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.625933 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef3f82de70bb4328a15a044fe0e500bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.626271 17798 tablet_service.cc:1511] Processing CreateTablet for tablet ef3f82de70bb4328a15a044fe0e500bf (DEFAULT_TABLE table=default.Table_cca06980f109487a9ed8b8172950b60d [id=016f4039733144c985808b862eebcc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.627390 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef3f82de70bb4328a15a044fe0e500bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.628587 17608 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_fa88730066c84caa82e692ad0e8bb34c" } new_table_name: "default.Table_56ddb56503844349ba42c68ebb67a694" modify_external_catalogs: true
I20260924 19:57:29.635260 17967 raft_consensus.cc:2468] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:29.639992 18223 ts_tablet_manager.cc:1434] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.090s user 0.006s sys 0.001s
I20260924 19:57:29.642792 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 916877c7540a4f6685064fc2807cb140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:29.643554 17753 leader_election.cc:304] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:29.644708 18348 ts_tablet_manager.cc:1929] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:29.645112 18348 log.cc:1199] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/1b9ffd3fc8714979bfaa24c36d6fe0f9
I20260924 19:57:29.645752 18348 ts_tablet_manager.cc:1950] T 1b9ffd3fc8714979bfaa24c36d6fe0f9 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:29.647607 18442 raft_consensus.cc:2804] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:29.648741 18442 raft_consensus.cc:697] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:29.649619 18442 consensus_queue.cc:237] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.655104 18220 tablet_bootstrap.cc:654] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:29.663712 18050 leader_election.cc:343] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:29.664196 18050 leader_election.cc:304] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:29.665017 18095 tablet_service.cc:1511] Processing CreateTablet for tablet ef3f82de70bb4328a15a044fe0e500bf (DEFAULT_TABLE table=default.Table_cca06980f109487a9ed8b8172950b60d [id=016f4039733144c985808b862eebcc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.665278 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 085b8088134d47478f0ce06a6a9f8572 (DEFAULT_TABLE table=default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.665684 18440 raft_consensus.cc:2749] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:29.666121 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef3f82de70bb4328a15a044fe0e500bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.666656 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 085b8088134d47478f0ce06a6a9f8572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.682511 18231 ts_tablet_manager.cc:1929] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:29.682933 18231 log.cc:1199] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/fb011ae3f182464ab68d77924ab70181
I20260924 19:57:29.686445 17947 tablet_service.cc:1511] Processing CreateTablet for tablet 085b8088134d47478f0ce06a6a9f8572 (DEFAULT_TABLE table=default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.687831 18223 tablet_bootstrap.cc:492] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:29.687884 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 085b8088134d47478f0ce06a6a9f8572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.695971 18449 tablet.cc:1751] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260924 19:57:29.700294 17802 tablet_service.cc:1511] Processing CreateTablet for tablet 9c2440e1cade4327a63d6234e3adcd64 (DEFAULT_TABLE table=default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.701329 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2440e1cade4327a63d6234e3adcd64. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:29.703949 17753 leader_election.cc:343] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 8a19b5812dc54589a5cbf48199951600
I20260924 19:57:29.706805 18403 consensus_queue.cc:1048] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:29.723760 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 44) took 1606 ms. Trace:
I20260924 19:57:29.724112 18348 rpcz_store.cc:276] 0924 19:57:28.117138 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:28.117289 (+ 151us) service_pool.cc:224] Handling call
0924 19:57:29.451143 (+1333854us) ts_tablet_manager.cc:1151] Deleting tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9
0924 19:57:29.451275 (+ 132us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:29.575806 (+124531us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.644584 (+ 68778us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.723698 (+ 79114us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1295955}
I20260924 19:57:29.725282 18096 tablet_service.cc:1511] Processing CreateTablet for tablet 085b8088134d47478f0ce06a6a9f8572 (DEFAULT_TABLE table=default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.725550 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 1b9ffd3fc8714979bfaa24c36d6fe0f9 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]) successfully deleted
I20260924 19:57:29.726339 18348 tablet_replica.cc:333] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
W20260924 19:57:29.726688 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet efd7339d0fee436299a7eac02a82addd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efd7339d0fee436299a7eac02a82addd already in progress: creating tablet
I20260924 19:57:29.728816 18220 tablet_bootstrap.cc:492] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:29.729192 18220 ts_tablet_manager.cc:1403] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.129s user 0.009s sys 0.000s
I20260924 19:57:29.731674 18231 ts_tablet_manager.cc:1950] T fb011ae3f182464ab68d77924ab70181 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:29.742681 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 48) took 1312 ms. Trace:
I20260924 19:57:29.743973 18220 raft_consensus.cc:359] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.744768 18220 raft_consensus.cc:385] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:29.745074 18220 raft_consensus.cc:740] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:29.748633 18348 raft_consensus.cc:2243] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:29.751677 18403 consensus_queue.cc:1048] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:29.751987 18348 raft_consensus.cc:2272] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:29.753000 17950 tablet_service.cc:1511] Processing CreateTablet for tablet 9c2440e1cade4327a63d6234e3adcd64 (DEFAULT_TABLE table=default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.754415 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2440e1cade4327a63d6234e3adcd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.760814 18411 raft_consensus.cc:493] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:29.761274 18411 raft_consensus.cc:515] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.763900 18411 leader_election.cc:290] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:29.754451 18348 ts_tablet_manager.cc:1916] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:29.768528 17602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.190:46309, user_credentials={real_user=slave}} blocked reactor thread for 73294.9us
I20260924 19:57:29.770557 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd2f548d5bd5454cb16335473a9eeb03" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:29.771567 18223 tablet_bootstrap.cc:654] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:29.745270 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 085b8088134d47478f0ce06a6a9f8572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.759022 18220 consensus_queue.cc:260] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.777608 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 9c2440e1cade4327a63d6234e3adcd64 (DEFAULT_TABLE table=default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.777868 17967 raft_consensus.cc:2468] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:29.778602 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c2440e1cade4327a63d6234e3adcd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.780979 18050 leader_election.cc:304] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:29.781664 18411 raft_consensus.cc:2804] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:29.781970 18411 raft_consensus.cc:493] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:29.782267 18411 raft_consensus.cc:3060] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:29.788707 18411 raft_consensus.cc:515] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.796248 18220 ts_tablet_manager.cc:1434] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.067s user 0.007s sys 0.000s
I20260924 19:57:29.743625 18224 tablet_bootstrap.cc:654] T d81f74c7ba2443e49228b17e867588cc P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:29.764452 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd2f548d5bd5454cb16335473a9eeb03" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:29.779368 17800 tablet_service.cc:1511] Processing CreateTablet for tablet c884f12860164f12bbcd0d2f87067013 (DEFAULT_TABLE table=default.Table_c9486a2257a84d9ab489210722683584 [id=54f05b78c3324978b2635be4bd969f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.803856 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c884f12860164f12bbcd0d2f87067013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.797186 17947 tablet_service.cc:1511] Processing CreateTablet for tablet c884f12860164f12bbcd0d2f87067013 (DEFAULT_TABLE table=default.Table_c9486a2257a84d9ab489210722683584 [id=54f05b78c3324978b2635be4bd969f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.805145 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c884f12860164f12bbcd0d2f87067013. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:29.807669 18051 leader_election.cc:343] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:29.811326 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd2f548d5bd5454cb16335473a9eeb03" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
W20260924 19:57:29.812565 18051 leader_election.cc:343] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:29.814755 18348 ts_tablet_manager.cc:1929] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:29.815145 18348 log.cc:1199] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/8a19b5812dc54589a5cbf48199951600
I20260924 19:57:29.815858 18348 ts_tablet_manager.cc:1950] T 8a19b5812dc54589a5cbf48199951600 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:29.817826 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 49) took 1074 ms. Trace:
I20260924 19:57:29.823837 18224 tablet_bootstrap.cc:492] T d81f74c7ba2443e49228b17e867588cc P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:29.824313 18224 ts_tablet_manager.cc:1403] T d81f74c7ba2443e49228b17e867588cc P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.230s user 0.013s sys 0.000s
I20260924 19:57:29.824632 18348 rpcz_store.cc:276] 0924 19:57:28.743370 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:28.743680 (+ 310us) service_pool.cc:224] Handling call
0924 19:57:29.726186 (+982506us) ts_tablet_manager.cc:1151] Deleting tablet 8a19b5812dc54589a5cbf48199951600
0924 19:57:29.726235 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:29.782990 (+ 56755us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.814631 (+ 31641us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.817773 (+ 3142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":285,"tablet-delete.queue_time_us":980979}
I20260924 19:57:29.829063 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 8a19b5812dc54589a5cbf48199951600 (table default.Table_37482fb26ba144619bb66861dceaab3b [id=06b8b073d6ae475c8125613b89463d3c]) successfully deleted
I20260924 19:57:29.829348 18348 tablet_replica.cc:333] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:29.831841 18348 raft_consensus.cc:2243] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:29.832582 18348 pending_rounds.cc:70] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: Trying to abort 1 pending ops.
I20260924 19:57:29.832870 18348 pending_rounds.cc:77] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986261842001920 op_type: NO_OP noop_request { }
I20260924 19:57:29.833825 18348 raft_consensus.cc:2889] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:29.827843 18224 raft_consensus.cc:359] T d81f74c7ba2443e49228b17e867588cc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.834869 18224 raft_consensus.cc:385] T d81f74c7ba2443e49228b17e867588cc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:29.822299 18220 tablet_bootstrap.cc:492] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:29.848048 18220 tablet_bootstrap.cc:654] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:29.848356 18224 raft_consensus.cc:740] T d81f74c7ba2443e49228b17e867588cc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:29.849300 18224 consensus_queue.cc:260] T d81f74c7ba2443e49228b17e867588cc P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.851598 18224 ts_tablet_manager.cc:1434] T d81f74c7ba2443e49228b17e867588cc P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260924 19:57:29.851660 18411 leader_election.cc:290] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:29.855062 18224 tablet_bootstrap.cc:492] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:29.855574 17615 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet a2e4519aed7642678db3a518a89778de
I20260924 19:57:29.857734 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_2b157ac766d244e081168b817296bd4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:29.860284 18224 tablet_bootstrap.cc:654] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:29.842196 18348 raft_consensus.cc:2272] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:29.850822 18449 tablet.cc:1746] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: Skipping requested alter to schema version 4, tablet already version 4
I20260924 19:57:29.864148 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet a2e4519aed7642678db3a518a89778de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:29.864202 17950 tablet_service.cc:1558] Processing DeleteTablet for tablet a2e4519aed7642678db3a518a89778de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:29.864997 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd2f548d5bd5454cb16335473a9eeb03" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:29.865571 17967 raft_consensus.cc:3060] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:29.866196 18348 ts_tablet_manager.cc:1916] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:29.868480 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet a2e4519aed7642678db3a518a89778de because the tablet was not found. No further retry: Not found: Tablet not found: a2e4519aed7642678db3a518a89778de
I20260924 19:57:29.871978 17967 raft_consensus.cc:2468] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:29.873332 18050 leader_election.cc:304] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
I20260924 19:57:29.875061 18440 raft_consensus.cc:2804] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:29.875627 18440 raft_consensus.cc:697] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:29.876438 18440 consensus_queue.cc:237] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
W20260924 19:57:29.877354 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b157ac766d244e081168b817296bd4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:29.892084 18220 tablet_bootstrap.cc:492] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:29.892491 18220 ts_tablet_manager.cc:1403] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.074s user 0.009s sys 0.000s
I20260924 19:57:29.895138 18220 raft_consensus.cc:359] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:29.895780 18220 raft_consensus.cc:385] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:29.896046 18220 raft_consensus.cc:740] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:29.899215 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet a2e4519aed7642678db3a518a89778de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:29.847644 18231 rpcz_store.cc:276] 0924 19:57:28.430534 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:28.430671 (+ 137us) service_pool.cc:224] Handling call
0924 19:57:29.338960 (+908289us) ts_tablet_manager.cc:1151] Deleting tablet fb011ae3f182464ab68d77924ab70181
0924 19:57:29.339015 (+ 55us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:29.550686 (+211671us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.682389 (+131703us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:29.741269 (+ 58880us) spinlock_profiling.cc:243] Waited 6.94 ms on lock 0x7b5800000f98. stack: 00007f7c8a13c323 00007f7c8a13c220 00007f7c89ce0574 0000000000516964 00000000005168f9 00007f7c89ff28a1 00007f7c8a136b51 00007f7c8a137182 00007f7c930a8ec9 00007f7c922bb918 00007f7c92382d82 00007f7c923b3960 00007f7c92399e6d 00007f7c92399e19 00007f7c92399da9 00007f7c92399d71
0924 19:57:29.742611 (+ 1342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15962368,"tablet-delete.queue_time_us":907618}
I20260924 19:57:29.899395 18220 consensus_queue.cc:260] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:29.922315 17611 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet dd2f548d5bd5454cb16335473a9eeb03
I20260924 19:57:29.923571 18220 ts_tablet_manager.cc:1434] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.031s user 0.003s sys 0.004s
I20260924 19:57:29.925009 18348 ts_tablet_manager.cc:1929] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:29.925268 17611 catalog_manager.cc:6934] ReplaceTablet: tablet dd2f548d5bd5454cb16335473a9eeb03 deleted and replaced by tablet 0e5f1e4d5655438e950d46a745cbb0db
I20260924 19:57:29.925371 18348 log.cc:1199] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/9b04059219084f608e8cb5fa4d0e0d58
I20260924 19:57:29.926024 18348 ts_tablet_manager.cc:1950] T 9b04059219084f608e8cb5fa4d0e0d58 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:29.926852 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 55) took 1287 ms. Trace:
I20260924 19:57:29.930373 18231 rpcz_store.cc:276] 0924 19:57:28.639789 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:28.640012 (+ 223us) service_pool.cc:224] Handling call
0924 19:57:29.926519 (+1286507us) ts_tablet_manager.cc:1151] Deleting tablet efd7339d0fee436299a7eac02a82addd
0924 19:57:29.926564 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:29.926798 (+ 234us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255218}
I20260924 19:57:29.931515 18231 tablet_replica.cc:333] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:29.932274 18231 raft_consensus.cc:2243] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:29.935074 17612 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_7c382a08aec546da8c1aa720956036c7 [id=31e1c3788d264823a12ecd9755ec2027]). Expected version 4 got 0
I20260924 19:57:29.935493 18096 tablet_service.cc:1558] Processing DeleteTablet for tablet dd2f548d5bd5454cb16335473a9eeb03 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e5f1e4d5655438e950d46a745cbb0db) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:29.935474 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet fb011ae3f182464ab68d77924ab70181 (table default.Table_3d0e38cde76d4300a129a04a04b5915d [id=2d056489f9314bb59c83e506d47ca5b8]) successfully deleted
I20260924 19:57:29.936168 18301 tablet_replica.cc:333] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
W20260924 19:57:29.936180 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet efd7339d0fee436299a7eac02a82addd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efd7339d0fee436299a7eac02a82addd already in progress: creating tablet
I20260924 19:57:29.936964 18301 raft_consensus.cc:2243] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:29.937606 18301 pending_rounds.cc:70] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: Trying to abort 1 pending ops.
I20260924 19:57:29.937867 18301 pending_rounds.cc:77] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986265102655488 op_type: NO_OP noop_request { }
I20260924 19:57:29.938210 18301 raft_consensus.cc:2889] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:29.938522 18301 raft_consensus.cc:2272] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:29.939211 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet dd2f548d5bd5454cb16335473a9eeb03 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e5f1e4d5655438e950d46a745cbb0db) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:29.939374 18097 tablet_service.cc:1511] Processing CreateTablet for tablet c884f12860164f12bbcd0d2f87067013 (DEFAULT_TABLE table=default.Table_c9486a2257a84d9ab489210722683584 [id=54f05b78c3324978b2635be4bd969f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.940379 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c884f12860164f12bbcd0d2f87067013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.939922 17614 catalog_manager.cc:5838] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:29.940930 18220 tablet_bootstrap.cc:492] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:29.941347 17614 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet a384c9a0bb65429ebb97fe0807c78773 (table default.Table_1a379ab4fc834d69865de7106e463200 [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 1 got 0
I20260924 19:57:29.942287 17610 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet a8cc802bf87f4d739175f93203478af8
I20260924 19:57:29.942852 18301 ts_tablet_manager.cc:1916] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:29.946801 18220 tablet_bootstrap.cc:654] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:29.947181 17610 catalog_manager.cc:6934] ReplaceTablet: tablet a8cc802bf87f4d739175f93203478af8 deleted and replaced by tablet a765abd702644f079bddffb3fc4238ef
I20260924 19:57:29.948386 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 9b04059219084f608e8cb5fa4d0e0d58 (table default.Table_850acd445f004facb3a20aaac28003e3 [id=37ed98d304b9467fa4213115e9fd4da5]) successfully deleted
I20260924 19:57:29.949681 17614 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_7c382a08aec546da8c1aa720956036c7 [id=31e1c3788d264823a12ecd9755ec2027]). Expected version 4 got 0
I20260924 19:57:29.949980 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet a8cc802bf87f4d739175f93203478af8 with delete_type TABLET_DATA_DELETED (replaced by tablet a765abd702644f079bddffb3fc4238ef) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:29.952818 18096 tablet_service.cc:1558] Processing DeleteTablet for tablet a8cc802bf87f4d739175f93203478af8 with delete_type TABLET_DATA_DELETED (replaced by tablet a765abd702644f079bddffb3fc4238ef) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:29.953557 18348 tablet_replica.cc:333] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:29.954227 18348 raft_consensus.cc:2243] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:29.954653 18348 raft_consensus.cc:2272] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:29.936769 18224 tablet_bootstrap.cc:492] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:29.958513 18224 ts_tablet_manager.cc:1403] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.104s user 0.010s sys 0.000s
I20260924 19:57:29.959038 17802 tablet_service.cc:1558] Processing DeleteTablet for tablet a8cc802bf87f4d739175f93203478af8 with delete_type TABLET_DATA_DELETED (replaced by tablet a765abd702644f079bddffb3fc4238ef) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:29.960824 18224 raft_consensus.cc:359] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.961423 18224 raft_consensus.cc:385] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:29.961676 18224 raft_consensus.cc:740] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:29.962266 18224 consensus_queue.cc:260] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:29.956975 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet a8cc802bf87f4d739175f93203478af8 with delete_type TABLET_DATA_DELETED (replaced by tablet a765abd702644f079bddffb3fc4238ef) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:29.956835 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet fb011ae3f182464ab68d77924ab70181 because the tablet was not found. No further retry: Not found: Tablet not found: fb011ae3f182464ab68d77924ab70181
I20260924 19:57:29.967450 18224 ts_tablet_manager.cc:1434] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260924 19:57:29.967763 18231 pending_rounds.cc:70] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: Trying to abort 1 pending ops.
I20260924 19:57:29.967988 18231 pending_rounds.cc:77] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986264175443968 op_type: NO_OP noop_request { }
I20260924 19:57:29.968312 18231 raft_consensus.cc:2889] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20260924 19:57:29.969337 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 916877c7540a4f6685064fc2807cb140 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 916877c7540a4f6685064fc2807cb140 already in progress: creating tablet
W20260924 19:57:29.970022 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet a2e4519aed7642678db3a518a89778de because the tablet was not found. No further retry: Not found: Tablet not found: a2e4519aed7642678db3a518a89778de
I20260924 19:57:29.970569 18224 tablet_bootstrap.cc:492] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:29.958062 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet dd2f548d5bd5454cb16335473a9eeb03 with delete_type TABLET_DATA_DELETED (replaced by tablet 0e5f1e4d5655438e950d46a745cbb0db) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:29.972203 18301 ts_tablet_manager.cc:1929] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:29.972043 17947 tablet_service.cc:1511] Processing CreateTablet for tablet 201ea0020cb3430eac85e80b8c06fb07 (DEFAULT_TABLE table=default.Table_fa88730066c84caa82e692ad0e8bb34c [id=522c4df3169f4d2ba1cbe15df69d40b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.972566 18301 log.cc:1199] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/dd2f548d5bd5454cb16335473a9eeb03
I20260924 19:57:29.933822 17798 tablet_service.cc:1511] Processing CreateTablet for tablet 201ea0020cb3430eac85e80b8c06fb07 (DEFAULT_TABLE table=default.Table_fa88730066c84caa82e692ad0e8bb34c [id=522c4df3169f4d2ba1cbe15df69d40b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.973222 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 201ea0020cb3430eac85e80b8c06fb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.973309 18301 ts_tablet_manager.cc:1950] T dd2f548d5bd5454cb16335473a9eeb03 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:29.973439 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 201ea0020cb3430eac85e80b8c06fb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.978058 17801 tablet_service.cc:1511] Processing CreateTablet for tablet f38188eccc344246b72a8755ec5201ed (DEFAULT_TABLE table=default.Table_37482fb26ba144619bb66861dceaab3b [id=06b8b073d6ae475c8125613b89463d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.978415 18348 ts_tablet_manager.cc:1916] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:29.981901 18223 tablet_bootstrap.cc:492] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:29.982357 18223 ts_tablet_manager.cc:1403] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.295s user 0.009s sys 0.001s
I20260924 19:57:29.984929 18223 raft_consensus.cc:359] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:29.986050 17600 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 954178b8cc1e434085337f66b0e85672 consensus state has no leader
I20260924 19:57:29.986698 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f38188eccc344246b72a8755ec5201ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:29.968631 18231 raft_consensus.cc:2272] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:29.990878 18224 tablet_bootstrap.cc:654] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:29.985625 18223 raft_consensus.cc:385] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:29.992676 18223 raft_consensus.cc:740] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:29.992985 18231 ts_tablet_manager.cc:1916] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:29.996008 18442 raft_consensus.cc:493] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:29.996613 18442 raft_consensus.cc:515] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
W20260924 19:57:30.002357 17753 leader_election.cc:343] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 9b04059219084f608e8cb5fa4d0e0d58
W20260924 19:57:30.005205 17753 leader_election.cc:343] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Not found: Tablet not found: 9b04059219084f608e8cb5fa4d0e0d58
I20260924 19:57:30.005718 17753 leader_election.cc:304] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:30.006529 18290 raft_consensus.cc:2749] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:30.008302 18442 leader_election.cc:290] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:30.009140 17950 tablet_service.cc:1511] Processing CreateTablet for tablet f38188eccc344246b72a8755ec5201ed (DEFAULT_TABLE table=default.Table_37482fb26ba144619bb66861dceaab3b [id=06b8b073d6ae475c8125613b89463d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.010241 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f38188eccc344246b72a8755ec5201ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.009951 17800 tablet_service.cc:1511] Processing CreateTablet for tablet a2e4519aed7642678db3a518a89778de (DEFAULT_TABLE table=default.Table_a4745efada6f4edea3f2d4bad1c3905e [id=993f4ef55cb44fb09a9c543e7cf1f2b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:29.975710 18301 tablet_replica.cc:333] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:30.004499 18096 tablet_service.cc:1511] Processing CreateTablet for tablet 201ea0020cb3430eac85e80b8c06fb07 (DEFAULT_TABLE table=default.Table_fa88730066c84caa82e692ad0e8bb34c [id=522c4df3169f4d2ba1cbe15df69d40b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.015975 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e4519aed7642678db3a518a89778de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.016192 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 201ea0020cb3430eac85e80b8c06fb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.014869 18223 consensus_queue.cc:260] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.023382 18223 ts_tablet_manager.cc:1434] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20260924 19:57:30.026626 18301 raft_consensus.cc:2243] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:30.027658 18301 raft_consensus.cc:2272] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:30.029812 17799 tablet_service.cc:1511] Processing CreateTablet for tablet 954178b8cc1e434085337f66b0e85672 (DEFAULT_TABLE table=default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.030872 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 954178b8cc1e434085337f66b0e85672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.029512 18095 tablet_service.cc:1511] Processing CreateTablet for tablet f38188eccc344246b72a8755ec5201ed (DEFAULT_TABLE table=default.Table_37482fb26ba144619bb66861dceaab3b [id=06b8b073d6ae475c8125613b89463d3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.033010 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f38188eccc344246b72a8755ec5201ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.036660 18224 tablet_bootstrap.cc:492] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:30.037032 18224 ts_tablet_manager.cc:1403] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.000s
I20260924 19:57:30.039356 18223 tablet_bootstrap.cc:492] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:30.039979 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet dd2f548d5bd5454cb16335473a9eeb03 (table default.Table_9740734f1493496d94dac20cb0e65c85 [id=0ceede10f5ea46729576babbd0df0c56]) successfully deleted
I20260924 19:57:30.041965 17949 tablet_service.cc:1511] Processing CreateTablet for tablet a2e4519aed7642678db3a518a89778de (DEFAULT_TABLE table=default.Table_a4745efada6f4edea3f2d4bad1c3905e [id=993f4ef55cb44fb09a9c543e7cf1f2b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.043010 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e4519aed7642678db3a518a89778de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.048441 17821 raft_consensus.cc:3060] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.048825 18301 ts_tablet_manager.cc:1916] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:30.051537 17613 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1684aaf9d984649bff8c4b0b12044b9 has no consensus state
I20260924 19:57:30.055686 18224 raft_consensus.cc:359] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.056327 18224 raft_consensus.cc:385] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.056569 18224 raft_consensus.cc:740] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.057168 18224 consensus_queue.cc:260] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.060228 18403 consensus_queue.cc:1048] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:30.069530 17608 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 201ea0020cb3430eac85e80b8c06fb07 consensus state has no leader
I20260924 19:57:30.070734 18223 tablet_bootstrap.cc:654] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.073948 17802 tablet_service.cc:1511] Processing CreateTablet for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (DEFAULT_TABLE table=default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.075052 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a524bad1c4040e58ad84fcf5a4fc47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.081851 18440 raft_consensus.cc:493] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:30.082342 18440 raft_consensus.cc:515] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.084408 18440 leader_election.cc:290] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:30.085909 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f482b53a4a2d44f6b3431c588cd605ce" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:30.086423 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f482b53a4a2d44f6b3431c588cd605ce" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:30.089088 18224 ts_tablet_manager.cc:1434] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.052s user 0.005s sys 0.000s
W20260924 19:57:30.089175 18050 leader_election.cc:343] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:30.090114 18051 leader_election.cc:343] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:30.090495 18051 leader_election.cc:304] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:30.092093 18440 raft_consensus.cc:2749] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:30.092469 18098 tablet_service.cc:1511] Processing CreateTablet for tablet a2e4519aed7642678db3a518a89778de (DEFAULT_TABLE table=default.Table_a4745efada6f4edea3f2d4bad1c3905e [id=993f4ef55cb44fb09a9c543e7cf1f2b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.093621 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2e4519aed7642678db3a518a89778de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.094980 17952 tablet_service.cc:1511] Processing CreateTablet for tablet 954178b8cc1e434085337f66b0e85672 (DEFAULT_TABLE table=default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.096135 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 954178b8cc1e434085337f66b0e85672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.085054 18220 tablet_bootstrap.cc:492] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:30.098213 18220 ts_tablet_manager.cc:1403] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.157s user 0.011s sys 0.006s
I20260924 19:57:30.109798 17612 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_9740734f1493496d94dac20cb0e65c85" } modify_external_catalogs: true
I20260924 19:57:30.109872 18220 raft_consensus.cc:359] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.110466 18220 raft_consensus.cc:385] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.110419 17612 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_9740734f1493496d94dac20cb0e65c85" } modify_external_catalogs: true
I20260924 19:57:30.110702 18220 raft_consensus.cc:740] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.111340 18220 consensus_queue.cc:260] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.115571 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_4e897947eb9e4466a9f050091b6a7b60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:30.117336 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e897947eb9e4466a9f050091b6a7b60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:30.130091 18094 tablet_service.cc:1511] Processing CreateTablet for tablet 954178b8cc1e434085337f66b0e85672 (DEFAULT_TABLE table=default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.131287 18094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 954178b8cc1e434085337f66b0e85672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.140882 17607 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_06a8017f949541c0bbe0df4972ebc15d" } modify_external_catalogs: true
I20260924 19:57:30.141453 17607 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_06a8017f949541c0bbe0df4972ebc15d" } modify_external_catalogs: true
I20260924 19:57:30.153573 18220 ts_tablet_manager.cc:1434] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.055s user 0.005s sys 0.000s
I20260924 19:57:30.162889 17798 tablet_service.cc:1511] Processing CreateTablet for tablet 151247d6b5254cd884c58db9311ac04a (DEFAULT_TABLE table=default.Table_c8e27940040c495780f92f92431541cb [id=47a10be86ad441a7961c0f34fed46040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.164191 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 151247d6b5254cd884c58db9311ac04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.164793 17951 tablet_service.cc:1511] Processing CreateTablet for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (DEFAULT_TABLE table=default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.165594 18224 tablet_bootstrap.cc:492] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:30.165755 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a524bad1c4040e58ad84fcf5a4fc47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.165182 17799 tablet_service.cc:1511] Processing CreateTablet for tablet dd434e38c9364f55bd18383149015028 (DEFAULT_TABLE table=default.Table_7190cd8a957a47368155eba08e1e37f2 [id=be19c62729a34ae1a15966662303a869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.172603 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd434e38c9364f55bd18383149015028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.178537 18231 ts_tablet_manager.cc:1929] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:30.183995 17948 tablet_service.cc:1511] Processing CreateTablet for tablet 151247d6b5254cd884c58db9311ac04a (DEFAULT_TABLE table=default.Table_c8e27940040c495780f92f92431541cb [id=47a10be86ad441a7961c0f34fed46040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.185120 17948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 151247d6b5254cd884c58db9311ac04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.186094 18301 ts_tablet_manager.cc:1929] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:30.186502 18301 log.cc:1199] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/a8cc802bf87f4d739175f93203478af8
I20260924 19:57:30.186669 18440 raft_consensus.cc:493] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:30.187139 18440 raft_consensus.cc:515] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.187713 18231 log.cc:1199] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/8a19b5812dc54589a5cbf48199951600
I20260924 19:57:30.188874 18231 ts_tablet_manager.cc:1950] T 8a19b5812dc54589a5cbf48199951600 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:30.189163 18440 leader_election.cc:290] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:30.190102 18348 ts_tablet_manager.cc:1929] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:30.190496 18348 log.cc:1199] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/dd2f548d5bd5454cb16335473a9eeb03
I20260924 19:57:30.191133 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 54) took 1551 ms. Trace:
I20260924 19:57:30.191948 18231 rpcz_store.cc:276] 0924 19:57:28.639428 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:28.639653 (+ 225us) service_pool.cc:224] Handling call
0924 19:57:29.931359 (+1291706us) ts_tablet_manager.cc:1151] Deleting tablet 8a19b5812dc54589a5cbf48199951600
0924 19:57:29.931404 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:30.009829 (+ 78425us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:30.178420 (+168591us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:30.191072 (+ 12652us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1259926}
I20260924 19:57:30.196843 18224 tablet_bootstrap.cc:654] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.197813 18301 ts_tablet_manager.cc:1950] T a8cc802bf87f4d739175f93203478af8 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:30.191094 18348 ts_tablet_manager.cc:1950] T dd2f548d5bd5454cb16335473a9eeb03 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:30.198285 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd7339d0fee436299a7eac02a82addd" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:30.199882 18050 leader_election.cc:343] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:30.208145 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (DEFAULT_TABLE table=default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.209288 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a524bad1c4040e58ad84fcf5a4fc47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.160907 18426 consensus_queue.cc:1048] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:30.206727 18348 tablet_replica.cc:333] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:30.221725 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]) successfully deleted
I20260924 19:57:30.224114 18095 tablet_service.cc:1511] Processing CreateTablet for tablet dd434e38c9364f55bd18383149015028 (DEFAULT_TABLE table=default.Table_7190cd8a957a47368155eba08e1e37f2 [id=be19c62729a34ae1a15966662303a869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.225498 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd434e38c9364f55bd18383149015028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.227578 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet dd2f548d5bd5454cb16335473a9eeb03 (table default.Table_9740734f1493496d94dac20cb0e65c85 [id=0ceede10f5ea46729576babbd0df0c56]) successfully deleted
I20260924 19:57:30.232627 18223 tablet_bootstrap.cc:492] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:30.233042 18223 ts_tablet_manager.cc:1403] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.209s user 0.004s sys 0.011s
I20260924 19:57:30.235744 18223 raft_consensus.cc:359] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.236411 18223 raft_consensus.cc:385] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.236672 18223 raft_consensus.cc:740] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.237344 18223 consensus_queue.cc:260] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.238938 18231 tablet_replica.cc:333] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:30.240839 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 151247d6b5254cd884c58db9311ac04a (DEFAULT_TABLE table=default.Table_c8e27940040c495780f92f92431541cb [id=47a10be86ad441a7961c0f34fed46040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.242461 18231 raft_consensus.cc:2243] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:30.242816 18223 ts_tablet_manager.cc:1434] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260924 19:57:30.244287 18231 raft_consensus.cc:2272] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:30.247697 18223 tablet_bootstrap.cc:492] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:30.249577 18231 ts_tablet_manager.cc:1916] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:30.250855 17615 master_service.cc:960] ReplaceTablet: received request to replace tablet 916877c7540a4f6685064fc2807cb140 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:30.257129 17615 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_7190cd8a957a47368155eba08e1e37f2" } new_table_name: "default.Table_b278e4873a924338b9b880b50266f306" modify_external_catalogs: true
I20260924 19:57:30.259656 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_01931a7dc8a94582ae8c31c6078b66c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:30.261456 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01931a7dc8a94582ae8c31c6078b66c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:30.262777 18223 tablet_bootstrap.cc:654] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.265394 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_79bc1edaa2024011a001b435d63329f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:30.268165 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79bc1edaa2024011a001b435d63329f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:30.273892 17613 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_1a379ab4fc834d69865de7106e463200" } new_table_name: "default.Table_102d9262505c41738c4c97fc11551bcb" modify_external_catalogs: true
I20260924 19:57:30.278119 18223 tablet_bootstrap.cc:492] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:30.278554 18223 ts_tablet_manager.cc:1403] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.003s
I20260924 19:57:30.291568 17946 tablet_service.cc:1511] Processing CreateTablet for tablet dd434e38c9364f55bd18383149015028 (DEFAULT_TABLE table=default.Table_7190cd8a957a47368155eba08e1e37f2 [id=be19c62729a34ae1a15966662303a869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:30.281282 18223 raft_consensus.cc:359] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.292318 18223 raft_consensus.cc:385] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.292584 18223 raft_consensus.cc:740] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.292937 17946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd434e38c9364f55bd18383149015028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.293545 18220 tablet_bootstrap.cc:492] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:30.293363 18223 consensus_queue.cc:260] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.295782 18223 ts_tablet_manager.cc:1434] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260924 19:57:30.299253 18223 tablet_bootstrap.cc:492] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:30.304873 18348 raft_consensus.cc:2243] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:30.305167 18220 tablet_bootstrap.cc:654] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.305524 18348 raft_consensus.cc:2272] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:30.308202 18223 tablet_bootstrap.cc:654] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.312381 18426 consensus_queue.cc:1048] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:30.329285 18348 ts_tablet_manager.cc:1916] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:30.334910 18426 raft_consensus.cc:493] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:30.335567 18426 raft_consensus.cc:515] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.337913 18426 leader_election.cc:290] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:30.372076 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f994e0401df4fc19bc5abb10f08bc6e" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:30.379261 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 151247d6b5254cd884c58db9311ac04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:30.379812 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 8a19b5812dc54589a5cbf48199951600 (table default.Table_37482fb26ba144619bb66861dceaab3b [id=06b8b073d6ae475c8125613b89463d3c]) successfully deleted
I20260924 19:57:30.383080 18117 raft_consensus.cc:2468] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:30.385035 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd7339d0fee436299a7eac02a82addd" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:30.385788 18231 ts_tablet_manager.cc:1929] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:30.385697 17823 raft_consensus.cc:2468] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:30.386770 18231 log.cc:1199] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/9b04059219084f608e8cb5fa4d0e0d58
I20260924 19:57:30.387534 18231 ts_tablet_manager.cc:1950] T 9b04059219084f608e8cb5fa4d0e0d58 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:30.387679 17822 raft_consensus.cc:3060] T d81f74c7ba2443e49228b17e867588cc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.395500 18051 leader_election.cc:304] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:30.396353 18440 raft_consensus.cc:2804] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:30.396684 18440 raft_consensus.cc:493] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:30.396684 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f994e0401df4fc19bc5abb10f08bc6e" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:30.397003 18440 raft_consensus.cc:3060] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.397461 17821 raft_consensus.cc:2468] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:30.399507 17906 leader_election.cc:304] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:30.400429 18413 raft_consensus.cc:2804] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:30.400750 18413 raft_consensus.cc:493] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:30.401087 18413 raft_consensus.cc:3060] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:30.401927 17599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.189:43689, user_credentials={real_user=slave}} blocked reactor thread for 173625us
I20260924 19:57:30.403654 18223 tablet_bootstrap.cc:492] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:30.404135 18223 ts_tablet_manager.cc:1403] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.105s user 0.027s sys 0.004s
I20260924 19:57:30.403699 18440 raft_consensus.cc:515] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.389523 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 58) took 1191 ms. Trace:
I20260924 19:57:30.406461 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd7339d0fee436299a7eac02a82addd" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:30.406338 18231 rpcz_store.cc:276] 0924 19:57:29.197771 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:29.197889 (+ 118us) service_pool.cc:224] Handling call
0924 19:57:30.238768 (+1040879us) ts_tablet_manager.cc:1151] Deleting tablet 9b04059219084f608e8cb5fa4d0e0d58
0924 19:57:30.238817 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:30.376875 (+138058us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:30.385664 (+ 8789us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:30.389465 (+ 3801us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1040294}
I20260924 19:57:30.407644 18231 tablet_replica.cc:333] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:30.407066 17821 raft_consensus.cc:3060] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.408360 18231 raft_consensus.cc:2243] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:30.408771 18231 raft_consensus.cc:2272] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:30.410422 18231 ts_tablet_manager.cc:1916] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:30.411926 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 9b04059219084f608e8cb5fa4d0e0d58 (table default.Table_850acd445f004facb3a20aaac28003e3 [id=37ed98d304b9467fa4213115e9fd4da5]) successfully deleted
I20260924 19:57:30.415002 18440 leader_election.cc:290] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:30.415763 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd7339d0fee436299a7eac02a82addd" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
W20260924 19:57:30.417327 18050 leader_election.cc:343] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:30.424415 18223 raft_consensus.cc:359] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.424717 18426 consensus_queue.cc:1048] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:30.426220 18220 tablet_bootstrap.cc:492] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:30.426724 18220 ts_tablet_manager.cc:1403] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.133s user 0.007s sys 0.010s
I20260924 19:57:30.431069 18220 raft_consensus.cc:359] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.432979 18223 raft_consensus.cc:385] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.433285 18223 raft_consensus.cc:740] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.435751 17821 raft_consensus.cc:2468] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:30.438220 18223 consensus_queue.cc:260] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.446944 18223 ts_tablet_manager.cc:1434] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20260924 19:57:30.447937 18223 tablet_bootstrap.cc:492] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:30.455687 17607 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet e19a1487bd0c47dba542f2c050b88191
I20260924 19:57:30.457926 17612 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 0e5f1e4d5655438e950d46a745cbb0db
I20260924 19:57:30.458570 17612 master_service.cc:960] ReplaceTablet: received request to replace tablet 085b8088134d47478f0ce06a6a9f8572 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:30.437356 18413 raft_consensus.cc:515] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.461321 18220 raft_consensus.cc:385] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.462322 18220 raft_consensus.cc:740] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.462733 18051 leader_election.cc:304] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:30.463798 18440 raft_consensus.cc:2804] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:30.465113 18224 tablet_bootstrap.cc:492] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:30.465546 18224 ts_tablet_manager.cc:1403] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.300s user 0.016s sys 0.001s
I20260924 19:57:30.470256 18440 raft_consensus.cc:697] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:30.472704 18440 consensus_queue.cc:237] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.467973 18224 raft_consensus.cc:359] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.475320 18224 raft_consensus.cc:385] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.475708 18224 raft_consensus.cc:740] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.476513 18224 consensus_queue.cc:260] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.484702 18223 tablet_bootstrap.cc:654] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.491662 18224 ts_tablet_manager.cc:1434] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260924 19:57:30.492889 18224 tablet_bootstrap.cc:492] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:30.512511 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f994e0401df4fc19bc5abb10f08bc6e" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:30.513048 18117 raft_consensus.cc:3060] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.513199 18348 ts_tablet_manager.cc:1929] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:30.513561 18348 log.cc:1199] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/a8cc802bf87f4d739175f93203478af8
I20260924 19:57:30.514281 18348 ts_tablet_manager.cc:1950] T a8cc802bf87f4d739175f93203478af8 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
W20260924 19:57:30.515743 17599 catalog_manager.cc:5136] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): alter failed for tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]),no further retry: Not found: Tablet not found: a8cc802bf87f4d739175f93203478af8
I20260924 19:57:30.518481 18224 tablet_bootstrap.cc:654] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.521492 18117 raft_consensus.cc:2468] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:30.522298 18413 leader_election.cc:290] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:30.506182 18220 consensus_queue.cc:260] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.524861 18220 ts_tablet_manager.cc:1434] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.098s user 0.003s sys 0.002s
I20260924 19:57:30.528882 18220 tablet_bootstrap.cc:492] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:30.529008 18231 ts_tablet_manager.cc:1929] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:30.534605 18224 tablet_bootstrap.cc:492] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:30.529452 18231 log.cc:1199] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/a9505fd426134d068b95639012112bf3
I20260924 19:57:30.535960 18231 ts_tablet_manager.cc:1950] T a9505fd426134d068b95639012112bf3 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:30.536705 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 57) took 1353 ms. Trace:
I20260924 19:57:30.535082 18224 ts_tablet_manager.cc:1403] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.004s
I20260924 19:57:30.536994 18231 rpcz_store.cc:276] 0924 19:57:29.197313 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:29.197865 (+ 552us) service_pool.cc:224] Handling call
0924 19:57:30.407496 (+1209631us) ts_tablet_manager.cc:1151] Deleting tablet a9505fd426134d068b95639012112bf3
0924 19:57:30.407539 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:30.437814 (+ 30275us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:30.528875 (+ 91061us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:30.536649 (+ 7774us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206045}
I20260924 19:57:30.540962 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f994e0401df4fc19bc5abb10f08bc6e" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:30.541550 17823 raft_consensus.cc:3060] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.546984 18220 tablet_bootstrap.cc:654] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.548553 18440 raft_consensus.cc:493] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:30.549041 18440 raft_consensus.cc:515] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.550081 18224 raft_consensus.cc:359] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.550799 18224 raft_consensus.cc:385] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.551867 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349c6e9413bb41268d0c41d549203e2a" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
W20260924 19:57:30.553457 18051 leader_election.cc:343] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:30.555032 17823 raft_consensus.cc:2468] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:30.557214 18440 leader_election.cc:290] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
W20260924 19:57:30.559252 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 916877c7540a4f6685064fc2807cb140 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 916877c7540a4f6685064fc2807cb140 already in progress: creating tablet
I20260924 19:57:30.551076 18224 raft_consensus.cc:740] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
W20260924 19:57:30.560856 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet a2e4519aed7642678db3a518a89778de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2e4519aed7642678db3a518a89778de already in progress: creating tablet
W20260924 19:57:30.561740 17599 catalog_manager.cc:5136] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): alter failed for tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]),no further retry: Not found: Tablet not found: a8cc802bf87f4d739175f93203478af8
I20260924 19:57:30.548321 17905 leader_election.cc:304] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
W20260924 19:57:30.562778 17615 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd434e38c9364f55bd18383149015028 consensus state has no leader
I20260924 19:57:30.563220 18231 tablet_replica.cc:333] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:30.563309 18413 raft_consensus.cc:2804] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:30.563757 18413 raft_consensus.cc:697] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:30.564220 18231 raft_consensus.cc:2243] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:30.564687 18231 raft_consensus.cc:2272] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:30.564327 18224 consensus_queue.cc:260] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.564530 18413 consensus_queue.cc:237] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.566484 18231 ts_tablet_manager.cc:1916] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:30.571893 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349c6e9413bb41268d0c41d549203e2a" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:30.574389 18050 leader_election.cc:343] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:30.574833 18050 leader_election.cc:304] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:30.576296 18440 raft_consensus.cc:2749] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:30.575949 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_2bb6cac7492d4c5ca461d126788c9722"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:30.582525 18440 raft_consensus.cc:493] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:30.583015 18440 raft_consensus.cc:515] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:30.583743 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2bb6cac7492d4c5ca461d126788c9722 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:30.585098 18440 leader_election.cc:290] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:30.587661 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916877c7540a4f6685064fc2807cb140" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
W20260924 19:57:30.589083 18051 leader_election.cc:343] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:30.592151 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916877c7540a4f6685064fc2807cb140" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:30.592317 17608 master_service.cc:960] ReplaceTablet: received request to replace tablet a384c9a0bb65429ebb97fe0807c78773 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:30.593655 18224 ts_tablet_manager.cc:1434] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20260924 19:57:30.596161 18223 tablet_bootstrap.cc:492] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:30.596570 18223 ts_tablet_manager.cc:1403] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.149s user 0.015s sys 0.002s
I20260924 19:57:30.598805 18223 raft_consensus.cc:359] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.599457 18223 raft_consensus.cc:385] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.599702 18223 raft_consensus.cc:740] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
W20260924 19:57:30.600741 18050 leader_election.cc:343] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:30.601142 18050 leader_election.cc:304] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:30.601850 18440 raft_consensus.cc:2749] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:30.604732 18224 tablet_bootstrap.cc:492] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:30.617511 18223 consensus_queue.cc:260] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.626684 18223 ts_tablet_manager.cc:1434] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20260924 19:57:30.631145 18220 tablet_bootstrap.cc:492] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:30.631584 18220 ts_tablet_manager.cc:1403] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.008s
I20260924 19:57:30.633841 18220 raft_consensus.cc:359] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.634447 18220 raft_consensus.cc:385] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.634693 18220 raft_consensus.cc:740] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.635432 18223 tablet_bootstrap.cc:492] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:30.638638 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]) successfully deleted
I20260924 19:57:30.640268 18231 ts_tablet_manager.cc:1929] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:30.636924 18220 consensus_queue.cc:260] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.643879 18231 log.cc:1199] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/a8cc802bf87f4d739175f93203478af8
I20260924 19:57:30.644675 18231 ts_tablet_manager.cc:1950] T a8cc802bf87f4d739175f93203478af8 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:30.647275 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet a8cc802bf87f4d739175f93203478af8 (table default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]) successfully deleted
I20260924 19:57:30.648373 18224 tablet_bootstrap.cc:654] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:30.648965 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet a8cc802bf87f4d739175f93203478af8 because the tablet was not found. No further retry: Not found: Tablet not found: a8cc802bf87f4d739175f93203478af8
I20260924 19:57:30.652137 18220 ts_tablet_manager.cc:1434] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.020s user 0.004s sys 0.001s
I20260924 19:57:30.659778 18220 tablet_bootstrap.cc:492] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:30.659554 17607 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_5f33677cfb6a4eab8497cae637ecdf26" } modify_external_catalogs: true
I20260924 19:57:30.660149 17607 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_5f33677cfb6a4eab8497cae637ecdf26" } modify_external_catalogs: true
I20260924 19:57:30.660414 18231 tablet_replica.cc:333] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:30.661116 18231 raft_consensus.cc:2243] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:30.661572 18231 raft_consensus.cc:2272] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:30.662182 18413 raft_consensus.cc:493] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:30.662343 17606 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_37482fb26ba144619bb66861dceaab3b" } modify_external_catalogs: true
I20260924 19:57:30.662840 17606 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_37482fb26ba144619bb66861dceaab3b" } modify_external_catalogs: true
I20260924 19:57:30.662636 18413 raft_consensus.cc:515] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.665264 18231 ts_tablet_manager.cc:1916] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:30.666958 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97a1b8ccd35a43cea7a71d705fad1494" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:30.667703 17823 raft_consensus.cc:2468] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:30.668862 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97a1b8ccd35a43cea7a71d705fad1494" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:30.669469 18117 raft_consensus.cc:2468] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:30.670238 18413 leader_election.cc:290] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:30.673120 18224 tablet_bootstrap.cc:492] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:30.671710 17906 leader_election.cc:304] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:30.673591 18224 ts_tablet_manager.cc:1403] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.002s
I20260924 19:57:30.675067 18413 raft_consensus.cc:2804] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:30.675547 18413 raft_consensus.cc:493] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:30.675900 18413 raft_consensus.cc:3060] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.679859 18440 raft_consensus.cc:493] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:30.680354 18440 raft_consensus.cc:515] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.680145 18224 raft_consensus.cc:359] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.681648 18224 raft_consensus.cc:385] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.682487 18440 leader_election.cc:290] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:30.685204 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef3f82de70bb4328a15a044fe0e500bf" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:30.685647 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef3f82de70bb4328a15a044fe0e500bf" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:30.686149 18224 raft_consensus.cc:740] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
W20260924 19:57:30.686676 18050 leader_election.cc:343] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:30.687186 18051 leader_election.cc:343] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:30.687613 18051 leader_election.cc:304] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:30.688390 18475 raft_consensus.cc:2749] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:30.686821 18224 consensus_queue.cc:260] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.697131 18224 ts_tablet_manager.cc:1434] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260924 19:57:30.698380 18231 ts_tablet_manager.cc:1929] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:30.698782 18231 log.cc:1199] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/dd2f548d5bd5454cb16335473a9eeb03
I20260924 19:57:30.699472 18231 ts_tablet_manager.cc:1950] T dd2f548d5bd5454cb16335473a9eeb03 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:30.702629 18224 tablet_bootstrap.cc:492] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:30.703326 18413 raft_consensus.cc:515] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.707002 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97a1b8ccd35a43cea7a71d705fad1494" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:30.707577 18117 raft_consensus.cc:3060] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.712052 18224 tablet_bootstrap.cc:654] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.715054 18117 raft_consensus.cc:2468] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:30.716332 17905 leader_election.cc:304] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:30.717200 18426 raft_consensus.cc:2804] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:30.717630 18426 raft_consensus.cc:697] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:30.718425 18426 consensus_queue.cc:237] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:30.726992 18223 tablet_bootstrap.cc:654] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.730824 18220 tablet_bootstrap.cc:654] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.734946 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97a1b8ccd35a43cea7a71d705fad1494" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:30.735589 17823 raft_consensus.cc:3060] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.737634 18224 tablet_bootstrap.cc:492] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:30.738044 18224 ts_tablet_manager.cc:1403] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.000s
I20260924 19:57:30.740500 18224 raft_consensus.cc:359] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.741125 18224 raft_consensus.cc:385] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.741377 18224 raft_consensus.cc:740] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.742738 17823 raft_consensus.cc:2468] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:30.752816 18413 leader_election.cc:290] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:30.756100 18223 tablet_bootstrap.cc:492] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:30.756584 18223 ts_tablet_manager.cc:1403] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.129s user 0.012s sys 0.000s
I20260924 19:57:30.756417 18224 consensus_queue.cc:260] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.762610 18223 raft_consensus.cc:359] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.767498 18224 ts_tablet_manager.cc:1434] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
I20260924 19:57:30.767447 18223 raft_consensus.cc:385] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.767918 18223 raft_consensus.cc:740] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.768606 18223 consensus_queue.cc:260] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.769614 18220 tablet_bootstrap.cc:492] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:30.770059 18220 ts_tablet_manager.cc:1403] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.005s
I20260924 19:57:30.770726 18223 ts_tablet_manager.cc:1434] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260924 19:57:30.771598 18223 tablet_bootstrap.cc:492] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:30.772482 18220 raft_consensus.cc:359] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.773099 18220 raft_consensus.cc:385] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.773348 18220 raft_consensus.cc:740] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.773948 18220 consensus_queue.cc:260] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.775869 18220 ts_tablet_manager.cc:1434] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260924 19:57:30.776710 18220 tablet_bootstrap.cc:492] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:30.783444 18224 tablet_bootstrap.cc:492] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:30.791275 18220 tablet_bootstrap.cc:654] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.797130 18223 tablet_bootstrap.cc:654] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.798005 18224 tablet_bootstrap.cc:654] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.804400 17612 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 085b8088134d47478f0ce06a6a9f8572
I20260924 19:57:30.806691 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet 085b8088134d47478f0ce06a6a9f8572 with delete_type TABLET_DATA_DELETED (replaced by tablet fec9b6ecfb384b528cf71c4cfb2fd4cb) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:30.810606 18224 tablet_bootstrap.cc:492] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:30.811175 18224 ts_tablet_manager.cc:1403] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.003s
I20260924 19:57:30.811579 18223 tablet_bootstrap.cc:492] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:30.812023 18223 ts_tablet_manager.cc:1403] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.000s
W20260924 19:57:30.812049 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 085b8088134d47478f0ce06a6a9f8572 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 085b8088134d47478f0ce06a6a9f8572 already in progress: creating tablet
I20260924 19:57:30.812893 17612 catalog_manager.cc:6934] ReplaceTablet: tablet 085b8088134d47478f0ce06a6a9f8572 deleted and replaced by tablet fec9b6ecfb384b528cf71c4cfb2fd4cb
I20260924 19:57:30.813771 18224 raft_consensus.cc:359] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.814590 18224 raft_consensus.cc:385] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.814409 18223 raft_consensus.cc:359] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.814859 18224 raft_consensus.cc:740] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.815121 18223 raft_consensus.cc:385] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.815438 18223 raft_consensus.cc:740] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.815789 17946 tablet_service.cc:1558] Processing DeleteTablet for tablet 085b8088134d47478f0ce06a6a9f8572 with delete_type TABLET_DATA_DELETED (replaced by tablet fec9b6ecfb384b528cf71c4cfb2fd4cb) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:30.815586 18224 consensus_queue.cc:260] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.816088 18223 consensus_queue.cc:260] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.818082 18223 ts_tablet_manager.cc:1434] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 19:57:30.818328 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet 085b8088134d47478f0ce06a6a9f8572 with delete_type TABLET_DATA_DELETED (replaced by tablet fec9b6ecfb384b528cf71c4cfb2fd4cb) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:30.818959 18223 tablet_bootstrap.cc:492] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:30.819890 18492 tablet_replica.cc:333] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:30.820997 18492 raft_consensus.cc:2243] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:30.822549 18492 raft_consensus.cc:2272] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:30.828502 18224 ts_tablet_manager.cc:1434] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260924 19:57:30.829226 18492 ts_tablet_manager.cc:1916] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:30.831615 18413 raft_consensus.cc:493] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:30.832034 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 085b8088134d47478f0ce06a6a9f8572 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 085b8088134d47478f0ce06a6a9f8572 already in progress: creating tablet
I20260924 19:57:30.832070 18413 raft_consensus.cc:515] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.834041 18413 leader_election.cc:290] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:30.835351 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c7b8422753a4746bde71fcb25409949" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:30.835985 17823 raft_consensus.cc:2468] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:30.837494 17906 leader_election.cc:304] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:30.840051 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c7b8422753a4746bde71fcb25409949" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:30.840724 18117 raft_consensus.cc:2468] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:30.842878 18224 tablet_bootstrap.cc:492] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:30.843348 18413 raft_consensus.cc:2804] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:30.843772 18413 raft_consensus.cc:493] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:30.844172 18413 raft_consensus.cc:3060] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.845023 18220 tablet_bootstrap.cc:492] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:30.845433 18220 ts_tablet_manager.cc:1403] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.069s user 0.012s sys 0.010s
I20260924 19:57:30.848842 18224 tablet_bootstrap.cc:654] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.853806 18223 tablet_bootstrap.cc:654] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.854844 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_899397c3fffc4ae4b4d59756c4ab73cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:30.856609 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_899397c3fffc4ae4b4d59756c4ab73cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:30.848356 18220 raft_consensus.cc:359] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.863530 18220 raft_consensus.cc:385] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.863823 18220 raft_consensus.cc:740] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.865414 18224 tablet_bootstrap.cc:492] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:30.865839 18224 ts_tablet_manager.cc:1403] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.003s
I20260924 19:57:30.868526 18224 raft_consensus.cc:359] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.869184 18224 raft_consensus.cc:385] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.869473 18224 raft_consensus.cc:740] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.870107 18224 consensus_queue.cc:260] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.864626 18220 consensus_queue.cc:260] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.879475 18220 ts_tablet_manager.cc:1434] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.034s user 0.008s sys 0.000s
I20260924 19:57:30.881441 18426 raft_consensus.cc:493] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:30.881944 18426 raft_consensus.cc:515] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.882809 18220 tablet_bootstrap.cc:492] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:30.884120 18426 leader_election.cc:290] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:30.885870 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916877c7540a4f6685064fc2807cb140" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:30.886547 17823 raft_consensus.cc:2468] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:30.886793 18224 ts_tablet_manager.cc:1434] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260924 19:57:30.887982 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916877c7540a4f6685064fc2807cb140" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:30.888572 18117 raft_consensus.cc:2468] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:30.888863 18220 tablet_bootstrap.cc:654] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.889660 17906 leader_election.cc:304] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:30.890923 18426 raft_consensus.cc:2804] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:30.891294 18426 raft_consensus.cc:493] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:30.891629 18426 raft_consensus.cc:3060] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.897017 18224 tablet_bootstrap.cc:492] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:30.896940 18492 ts_tablet_manager.cc:1929] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:30.897356 18492 log.cc:1199] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/085b8088134d47478f0ce06a6a9f8572
I20260924 19:57:30.902526 18492 ts_tablet_manager.cc:1950] T 085b8088134d47478f0ce06a6a9f8572 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:30.904431 18224 tablet_bootstrap.cc:654] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.905330 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 085b8088134d47478f0ce06a6a9f8572 (table default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]) successfully deleted
I20260924 19:57:30.907310 18223 tablet_bootstrap.cc:492] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:30.907734 18223 ts_tablet_manager.cc:1403] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.089s user 0.014s sys 0.001s
I20260924 19:57:30.911119 18223 raft_consensus.cc:359] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.912091 18223 raft_consensus.cc:385] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.912451 18223 raft_consensus.cc:740] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.913612 17607 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:30.913331 18223 consensus_queue.cc:260] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.915023 18290 raft_consensus.cc:493] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:30.915539 18290 raft_consensus.cc:515] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.916687 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:30.917572 18290 leader_election.cc:290] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:30.919508 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 24f84032bca24a7ca9da65f0bbceb0d5 because the tablet was not found. No further retry: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:30.920478 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f482b53a4a2d44f6b3431c588cd605ce" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:30.921141 18117 raft_consensus.cc:2468] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:30.922313 17753 leader_election.cc:304] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:30.923043 18290 raft_consensus.cc:2804] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:30.923506 18224 tablet_bootstrap.cc:492] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:30.923436 18290 raft_consensus.cc:493] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:30.923892 18290 raft_consensus.cc:3060] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.924136 18224 ts_tablet_manager.cc:1403] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.003s
I20260924 19:57:30.927217 18224 raft_consensus.cc:359] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.927883 18224 raft_consensus.cc:385] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.928171 18224 raft_consensus.cc:740] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.928843 18224 consensus_queue.cc:260] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.933605 17607 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_2be07cf4cdd64ac5a99db4037ee35542" } new_table_name: "default.Table_41c56bb86db34f16aba6e94147cde1d2" modify_external_catalogs: true
I20260924 19:57:30.935887 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f482b53a4a2d44f6b3431c588cd605ce" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:30.936605 17967 raft_consensus.cc:2468] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:30.939133 18290 raft_consensus.cc:515] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.939790 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:30.922662 18223 ts_tablet_manager.cc:1434] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.015s user 0.003s sys 0.004s
I20260924 19:57:30.929004 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:30.945827 18223 tablet_bootstrap.cc:492] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:30.947484 18403 raft_consensus.cc:493] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:30.947906 18403 raft_consensus.cc:515] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.949764 18403 leader_election.cc:290] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:30.950891 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 24f84032bca24a7ca9da65f0bbceb0d5 because the tablet was not found. No further retry: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5
W20260924 19:57:30.951586 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 24f84032bca24a7ca9da65f0bbceb0d5 because the tablet was not found. No further retry: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:30.953195 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd7339d0fee436299a7eac02a82addd" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:30.953853 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f482b53a4a2d44f6b3431c588cd605ce" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:30.954535 17967 raft_consensus.cc:3060] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.955698 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd7339d0fee436299a7eac02a82addd" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:30.956525 17823 raft_consensus.cc:2393] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:30.957684 18220 tablet_bootstrap.cc:492] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:30.957731 17906 leader_election.cc:304] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:30.959270 18403 raft_consensus.cc:3060] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.960027 18426 raft_consensus.cc:515] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.961995 18426 leader_election.cc:290] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:30.958148 18220 ts_tablet_manager.cc:1403] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.076s user 0.016s sys 0.014s
I20260924 19:57:30.964419 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916877c7540a4f6685064fc2807cb140" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:30.964941 18117 raft_consensus.cc:3060] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.964895 18290 leader_election.cc:290] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:30.965276 18220 raft_consensus.cc:359] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.965876 18220 raft_consensus.cc:385] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:30.966113 18220 raft_consensus.cc:740] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:30.967568 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f482b53a4a2d44f6b3431c588cd605ce" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:30.968252 18118 raft_consensus.cc:3060] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.970691 18220 consensus_queue.cc:260] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.972625 18220 ts_tablet_manager.cc:1434] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260924 19:57:30.976953 18117 raft_consensus.cc:2468] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:30.977577 18220 tablet_bootstrap.cc:492] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:30.978039 18224 ts_tablet_manager.cc:1434] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.054s user 0.005s sys 0.020s
I20260924 19:57:30.980463 17967 raft_consensus.cc:2468] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:30.987749 18220 tablet_bootstrap.cc:654] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:30.990499 17753 leader_election.cc:304] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:30.991402 18498 raft_consensus.cc:2804] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:30.993402 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "916877c7540a4f6685064fc2807cb140" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:30.994035 17823 raft_consensus.cc:3060] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:30.991919 18498 raft_consensus.cc:697] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:30.995723 18498 consensus_queue.cc:237] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.990113 18224 tablet_bootstrap.cc:492] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:31.003846 18220 tablet_bootstrap.cc:492] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:31.004251 18220 ts_tablet_manager.cc:1403] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.004s
I20260924 19:57:31.005496 18403 raft_consensus.cc:2749] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:31.007191 17905 leader_election.cc:304] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:30.981845 18118 raft_consensus.cc:2468] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:30.989385 18223 tablet_bootstrap.cc:654] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.008960 18224 tablet_bootstrap.cc:654] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.009454 18220 raft_consensus.cc:359] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.010056 18220 raft_consensus.cc:385] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.010293 18220 raft_consensus.cc:740] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.012996 18220 consensus_queue.cc:260] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.014858 18426 raft_consensus.cc:2804] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:31.024111 17610 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet a384c9a0bb65429ebb97fe0807c78773 (table default.Table_1a379ab4fc834d69865de7106e463200 [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 1 got 0
I20260924 19:57:31.028383 18220 ts_tablet_manager.cc:1434] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
W20260924 19:57:31.035924 17600 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 954178b8cc1e434085337f66b0e85672 consensus state has no leader
I20260924 19:57:31.039201 18223 tablet_bootstrap.cc:492] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:31.039685 18223 ts_tablet_manager.cc:1403] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.094s user 0.019s sys 0.000s
I20260924 19:57:31.040490 18220 tablet_bootstrap.cc:492] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:31.042369 18223 raft_consensus.cc:359] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:30.851888 18413 raft_consensus.cc:515] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.043016 18223 raft_consensus.cc:385] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.043417 18223 raft_consensus.cc:740] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.044303 18223 consensus_queue.cc:260] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.046016 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c7b8422753a4746bde71fcb25409949" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:31.046687 17821 raft_consensus.cc:3060] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.048349 18224 tablet_bootstrap.cc:492] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:31.048817 18224 ts_tablet_manager.cc:1403] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.015s
I20260924 19:57:31.051627 18224 raft_consensus.cc:359] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.052356 18224 raft_consensus.cc:385] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.052656 18224 raft_consensus.cc:740] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.053013 18220 tablet_bootstrap.cc:654] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.054822 17821 raft_consensus.cc:2468] T 7c7b8422753a4746bde71fcb25409949 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:31.056809 18223 ts_tablet_manager.cc:1434] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260924 19:57:31.057243 18224 consensus_queue.cc:260] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.057750 18223 tablet_bootstrap.cc:492] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:31.061480 18224 ts_tablet_manager.cc:1434] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.012s user 0.007s sys 0.001s
I20260924 19:57:31.064558 18223 tablet_bootstrap.cc:654] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.065762 18498 raft_consensus.cc:493] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:31.066238 18498 raft_consensus.cc:515] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.070261 18220 tablet_bootstrap.cc:492] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:31.070704 18220 ts_tablet_manager.cc:1403] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20260924 19:57:31.072917 18223 tablet_bootstrap.cc:492] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:31.072739 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "085b8088134d47478f0ce06a6a9f8572" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:31.073311 18223 ts_tablet_manager.cc:1403] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20260924 19:57:31.073379 17967 raft_consensus.cc:2468] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:31.074309 17906 leader_election.cc:304] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:31.074769 17753 leader_election.cc:304] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:31.075569 18290 raft_consensus.cc:2804] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:31.075915 18290 raft_consensus.cc:493] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:31.076238 18290 raft_consensus.cc:3060] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.076870 18223 raft_consensus.cc:359] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.077550 18223 raft_consensus.cc:385] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.077829 18223 raft_consensus.cc:740] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.078500 18223 consensus_queue.cc:260] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:31.081221 17753 leader_election.cc:343] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 085b8088134d47478f0ce06a6a9f8572
I20260924 19:57:31.068423 18498 leader_election.cc:290] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.081653 18220 raft_consensus.cc:359] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.082335 18220 raft_consensus.cc:385] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.082612 18220 raft_consensus.cc:740] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.083314 18220 consensus_queue.cc:260] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.086897 18223 ts_tablet_manager.cc:1434] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.013s user 0.003s sys 0.004s
I20260924 19:57:31.087841 18223 tablet_bootstrap.cc:492] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:31.089612 18290 raft_consensus.cc:515] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.095439 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "085b8088134d47478f0ce06a6a9f8572" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:31.096242 17967 raft_consensus.cc:3060] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.096522 18223 tablet_bootstrap.cc:654] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.105707 17967 raft_consensus.cc:2468] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:31.106281 18224 tablet_bootstrap.cc:492] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:31.110455 18290 leader_election.cc:290] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:31.113934 17753 leader_election.cc:343] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 085b8088134d47478f0ce06a6a9f8572
I20260924 19:57:31.120383 17753 leader_election.cc:304] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters: eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:31.121917 18290 raft_consensus.cc:2804] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 19:57:31.122375 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 48193.9us
I20260924 19:57:31.125411 18403 raft_consensus.cc:2804] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:31.124878 18290 raft_consensus.cc:697] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:31.126068 18403 raft_consensus.cc:697] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:31.130541 18413 leader_election.cc:290] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.132104 18290 consensus_queue.cc:237] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.133124 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c7b8422753a4746bde71fcb25409949" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:31.134109 18117 raft_consensus.cc:3060] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.117690 18224 tablet_bootstrap.cc:654] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.148509 18224 tablet_bootstrap.cc:492] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:31.149085 18224 ts_tablet_manager.cc:1403] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.001s
W20260924 19:57:31.162158 18013 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad0a0 after lost signal to thread 17886
W20260924 19:57:31.163437 18013 debug-util.cc:398] Leaking SignalData structure 0x7b0800082c20 after lost signal to thread 18016
I20260924 19:57:31.168766 17671 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:31.172219 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:31.173904 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 24f84032bca24a7ca9da65f0bbceb0d5 because the tablet was not found. No further retry: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:31.174117 18224 raft_consensus.cc:359] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.174945 18224 raft_consensus.cc:385] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.175729 18224 raft_consensus.cc:740] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.176852 18224 consensus_queue.cc:260] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.179935 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:31.181468 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 24f84032bca24a7ca9da65f0bbceb0d5 because the tablet was not found. No further retry: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:31.189675 18224 ts_tablet_manager.cc:1434] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.040s user 0.006s sys 0.002s
I20260924 19:57:31.190537 17823 raft_consensus.cc:2468] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:31.199630 17611 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 97a1b8ccd35a43cea7a71d705fad1494 (table default.Table_850e5158d78d4850b87445dac75ef7a9 [id=f768ce157a7b407d9f1e55a5b9d0720a]). Expected version 1 got 0
I20260924 19:57:31.200522 17611 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 5f994e0401df4fc19bc5abb10f08bc6e (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]). Expected version 1 got 0
I20260924 19:57:31.201934 18426 raft_consensus.cc:697] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:31.202744 17611 catalog_manager.cc:5838] T d81f74c7ba2443e49228b17e867588cc P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:31.202831 18426 consensus_queue.cc:237] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.207572 17608 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet a384c9a0bb65429ebb97fe0807c78773
I20260924 19:57:31.209446 17608 catalog_manager.cc:6934] ReplaceTablet: tablet a384c9a0bb65429ebb97fe0807c78773 deleted and replaced by tablet 077f70bcee7e412299da12ddc30a63ff
I20260924 19:57:31.217358 17946 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:31.225013 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet a384c9a0bb65429ebb97fe0807c78773 with delete_type TABLET_DATA_DELETED (replaced by tablet 077f70bcee7e412299da12ddc30a63ff) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:31.225984 18492 tablet_replica.cc:333] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:31.227146 18492 raft_consensus.cc:2243] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:31.227897 18492 raft_consensus.cc:2272] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:31.231001 18492 ts_tablet_manager.cc:1916] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:31.238524 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet a384c9a0bb65429ebb97fe0807c78773 with delete_type TABLET_DATA_DELETED (replaced by tablet 077f70bcee7e412299da12ddc30a63ff) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:31.239545 18348 tablet_replica.cc:333] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
W20260924 19:57:31.241613 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 24f84032bca24a7ca9da65f0bbceb0d5 because the tablet was not found. No further retry: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:31.252830 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_a50429edeaf645668bc84244412bb359"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:31.255452 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a50429edeaf645668bc84244412bb359 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:31.260612 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_5e50f3dbbb674d658a5afac0ebcedaa6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:31.262456 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e50f3dbbb674d658a5afac0ebcedaa6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:31.275269 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 (DEFAULT_TABLE table=default.Table_5f33677cfb6a4eab8497cae637ecdf26 [id=6e11b13ce19a4d12893da7865efea4b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.276667 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f84032bca24a7ca9da65f0bbceb0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:31.291028 17952 tablet_service.cc:1511] Processing CreateTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 (DEFAULT_TABLE table=default.Table_5f33677cfb6a4eab8497cae637ecdf26 [id=6e11b13ce19a4d12893da7865efea4b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.292505 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f84032bca24a7ca9da65f0bbceb0d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:31.310137 17905 consensus_peers.cc:597] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:31.201114 18224 tablet_bootstrap.cc:492] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:31.213533 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet a384c9a0bb65429ebb97fe0807c78773 with delete_type TABLET_DATA_DELETED (replaced by tablet 077f70bcee7e412299da12ddc30a63ff) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:31.249686 18290 raft_consensus.cc:493] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:31.318395 18290 raft_consensus.cc:515] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.321321 18290 leader_election.cc:290] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.273949 17798 tablet_service.cc:1511] Processing CreateTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 (DEFAULT_TABLE table=default.Table_5f33677cfb6a4eab8497cae637ecdf26 [id=6e11b13ce19a4d12893da7865efea4b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.324857 18231 tablet_replica.cc:333] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:31.325652 18231 raft_consensus.cc:2243] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:31.326267 18231 raft_consensus.cc:2272] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:31.327906 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f84032bca24a7ca9da65f0bbceb0d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:31.327592 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349c6e9413bb41268d0c41d549203e2a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:31.328362 17967 raft_consensus.cc:2468] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:31.331733 17753 leader_election.cc:304] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:31.333158 18224 tablet_bootstrap.cc:654] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.340499 18231 ts_tablet_manager.cc:1916] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:31.347405 18290 raft_consensus.cc:2804] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:31.348032 18290 raft_consensus.cc:493] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:31.348500 18290 raft_consensus.cc:3060] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.349494 18224 tablet_bootstrap.cc:492] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:31.350153 18224 ts_tablet_manager.cc:1403] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.149s user 0.005s sys 0.009s
I20260924 19:57:31.353196 18224 raft_consensus.cc:359] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.353991 18224 raft_consensus.cc:385] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.354302 18224 raft_consensus.cc:740] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.360478 18224 consensus_queue.cc:260] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.363618 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349c6e9413bb41268d0c41d549203e2a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:31.364511 18118 raft_consensus.cc:2468] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:31.371300 18440 consensus_queue.cc:1048] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:31.375675 18224 ts_tablet_manager.cc:1434] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260924 19:57:31.379612 18440 raft_consensus.cc:493] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:31.380088 18440 raft_consensus.cc:515] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.382994 18440 leader_election.cc:290] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:31.385231 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "201ea0020cb3430eac85e80b8c06fb07" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:31.387565 18224 tablet_bootstrap.cc:492] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:31.386795 18050 leader_election.cc:343] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:31.389210 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "201ea0020cb3430eac85e80b8c06fb07" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
W20260924 19:57:31.390741 18051 leader_election.cc:343] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:31.391217 18051 leader_election.cc:304] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:31.392745 18440 raft_consensus.cc:2749] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:31.399672 18224 tablet_bootstrap.cc:654] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.418219 18440 consensus_queue.cc:1048] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:31.127270 18403 consensus_queue.cc:237] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.424582 18224 tablet_bootstrap.cc:492] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:31.425122 18224 ts_tablet_manager.cc:1403] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.000s
I20260924 19:57:31.428903 18224 raft_consensus.cc:359] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.429790 18224 raft_consensus.cc:385] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.434643 18224 raft_consensus.cc:740] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.435529 18224 consensus_queue.cc:260] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.444585 18224 ts_tablet_manager.cc:1434] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.019s user 0.006s sys 0.002s
W20260924 19:57:31.408597 17906 consensus_peers.cc:597] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:31.447320 18224 tablet_bootstrap.cc:492] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:31.431710 18348 raft_consensus.cc:2243] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:31.460409 18348 pending_rounds.cc:70] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: Trying to abort 1 pending ops.
I20260924 19:57:31.460853 18348 pending_rounds.cc:77] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332986270291488768 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "a384c9a0bb65429ebb97fe0807c78773" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_1a379ab4fc834d69865de7106e463200" dest_uuid: "f74e6ed351284d7fb110e44336ce643a" new_extra_config { } }
W20260924 19:57:31.462616 18348 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51332: Aborted: Op aborted
I20260924 19:57:31.464111 18348 raft_consensus.cc:2272] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:31.470618 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 62161.7us
I20260924 19:57:31.483460 18224 tablet_bootstrap.cc:654] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.486229 18223 tablet_bootstrap.cc:492] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:31.486824 18223 ts_tablet_manager.cc:1403] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.399s user 0.070s sys 0.099s
I20260924 19:57:31.489992 18223 raft_consensus.cc:359] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.490856 18223 raft_consensus.cc:385] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:57:31.500571 17599 catalog_manager.cc:5142] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): alter failed for tablet a384c9a0bb65429ebb97fe0807c78773 (table default.Table_1a379ab4fc834d69865de7106e463200 [id=3ee2ed8b290747128a29f7c107dbfe82]): Aborted: Op aborted
I20260924 19:57:31.474125 18348 ts_tablet_manager.cc:1916] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:31.504827 17606 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet f38188eccc344246b72a8755ec5201ed
I20260924 19:57:31.509147 18224 tablet_bootstrap.cc:492] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:31.509656 18224 ts_tablet_manager.cc:1403] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.008s
I20260924 19:57:31.512658 18224 raft_consensus.cc:359] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.513056 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet f38188eccc344246b72a8755ec5201ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:31.513373 18224 raft_consensus.cc:385] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.513682 18224 raft_consensus.cc:740] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.514446 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet f38188eccc344246b72a8755ec5201ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:31.504475 18220 ts_tablet_manager.cc:1434] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.433s user 0.067s sys 0.107s
I20260924 19:57:31.514578 18224 consensus_queue.cc:260] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.515933 18220 tablet_bootstrap.cc:492] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:31.519114 18224 ts_tablet_manager.cc:1434] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260924 19:57:31.520721 18224 tablet_bootstrap.cc:492] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:31.524556 18220 tablet_bootstrap.cc:654] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.491119 18223 raft_consensus.cc:740] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.528074 18223 consensus_queue.cc:260] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.531339 18224 tablet_bootstrap.cc:654] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.555872 18224 tablet_bootstrap.cc:492] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:31.556352 18224 ts_tablet_manager.cc:1403] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.000s
I20260924 19:57:31.559976 18223 ts_tablet_manager.cc:1434] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.073s user 0.010s sys 0.003s
I20260924 19:57:31.560957 18223 tablet_bootstrap.cc:492] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:31.581837 17799 tablet_service.cc:1511] Processing CreateTablet for tablet 0cf2df461598411f8ad9f3a51eafc6f7 (DEFAULT_TABLE table=default.Table_a10031a0fa004ac3ad48e6020477494b [id=7aab4014a5bc4823bbe845c021153fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.582861 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf2df461598411f8ad9f3a51eafc6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:31.586359 18426 raft_consensus.cc:493] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:31.586859 18426 raft_consensus.cc:515] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.589210 18426 leader_election.cc:290] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.591331 18224 raft_consensus.cc:359] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.594655 18224 raft_consensus.cc:385] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.595400 18224 raft_consensus.cc:740] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.597400 17802 tablet_service.cc:1511] Processing CreateTablet for tablet 910d61b8c3574302b5a328187226571e (DEFAULT_TABLE table=default.Table_2b157ac766d244e081168b817296bd4b [id=1bef6766d6a049a9a5126339c3f67842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.598754 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 910d61b8c3574302b5a328187226571e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:31.601159 18224 consensus_queue.cc:260] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.604453 17946 tablet_service.cc:1558] Processing DeleteTablet for tablet f38188eccc344246b72a8755ec5201ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:31.605351 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2440e1cade4327a63d6234e3adcd64" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:31.606688 18223 tablet_bootstrap.cc:654] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.606914 18498 raft_consensus.cc:493] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:31.611459 18498 raft_consensus.cc:515] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.616503 18224 ts_tablet_manager.cc:1434] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.060s user 0.008s sys 0.000s
I20260924 19:57:31.617987 18224 tablet_bootstrap.cc:492] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:31.619553 18498 leader_election.cc:290] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.621807 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef3f82de70bb4328a15a044fe0e500bf" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:31.622484 17967 raft_consensus.cc:2468] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:31.625567 17753 leader_election.cc:304] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:31.626983 18498 raft_consensus.cc:2804] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:31.627446 18498 raft_consensus.cc:493] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:31.627840 18498 raft_consensus.cc:3060] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.637193 18224 tablet_bootstrap.cc:654] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.652977 17952 tablet_service.cc:1511] Processing CreateTablet for tablet 0cf2df461598411f8ad9f3a51eafc6f7 (DEFAULT_TABLE table=default.Table_a10031a0fa004ac3ad48e6020477494b [id=7aab4014a5bc4823bbe845c021153fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.654068 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf2df461598411f8ad9f3a51eafc6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:31.659081 17823 raft_consensus.cc:2468] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:31.406853 18290 raft_consensus.cc:515] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.664935 18224 tablet_bootstrap.cc:492] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:31.665442 18224 ts_tablet_manager.cc:1403] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.001s
I20260924 19:57:31.669178 18224 raft_consensus.cc:359] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.669909 18224 raft_consensus.cc:385] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.670186 18224 raft_consensus.cc:740] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.670974 18224 consensus_queue.cc:260] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.678308 18220 tablet_bootstrap.cc:492] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:31.677884 18498 raft_consensus.cc:515] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.678766 18220 ts_tablet_manager.cc:1403] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.163s user 0.023s sys 0.011s
I20260924 19:57:31.681437 18220 raft_consensus.cc:359] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.682138 18220 raft_consensus.cc:385] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.682440 18220 raft_consensus.cc:740] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.683130 18220 consensus_queue.cc:260] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.685349 18224 ts_tablet_manager.cc:1434] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.020s user 0.007s sys 0.001s
I20260924 19:57:31.689460 18492 ts_tablet_manager.cc:1929] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:31.693158 18224 tablet_bootstrap.cc:492] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:31.701205 18224 tablet_bootstrap.cc:654] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.701997 17946 tablet_service.cc:1511] Processing CreateTablet for tablet 910d61b8c3574302b5a328187226571e (DEFAULT_TABLE table=default.Table_2b157ac766d244e081168b817296bd4b [id=1bef6766d6a049a9a5126339c3f67842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.703107 17946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 910d61b8c3574302b5a328187226571e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:31.708607 18220 ts_tablet_manager.cc:1434] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.030s user 0.003s sys 0.004s
I20260924 19:57:31.709594 18220 tablet_bootstrap.cc:492] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
W20260924 19:57:31.710896 17607 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 349c6e9413bb41268d0c41d549203e2a consensus state has no leader
I20260924 19:57:31.725409 18220 tablet_bootstrap.cc:654] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.726437 17906 leader_election.cc:304] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:31.727221 18508 raft_consensus.cc:2804] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:31.727571 18508 raft_consensus.cc:493] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:31.727887 18508 raft_consensus.cc:3060] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.736730 18508 raft_consensus.cc:515] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.739264 18224 tablet_bootstrap.cc:492] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:31.739782 18224 ts_tablet_manager.cc:1403] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.006s
I20260924 19:57:31.744557 18290 leader_election.cc:290] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.744432 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349c6e9413bb41268d0c41d549203e2a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:31.745015 17967 raft_consensus.cc:3060] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.742575 18224 raft_consensus.cc:359] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.746948 18220 tablet_bootstrap.cc:492] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:31.747243 18224 raft_consensus.cc:385] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.747455 18220 ts_tablet_manager.cc:1403] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.001s
I20260924 19:57:31.747541 18224 raft_consensus.cc:740] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.749958 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef3f82de70bb4328a15a044fe0e500bf" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:31.750487 17972 raft_consensus.cc:3060] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.750554 18220 raft_consensus.cc:359] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.751302 18220 raft_consensus.cc:385] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.751574 18220 raft_consensus.cc:740] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.752272 18220 consensus_queue.cc:260] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.753471 18508 leader_election.cc:290] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.753494 17967 raft_consensus.cc:2468] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:31.754442 18220 ts_tablet_manager.cc:1434] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260924 19:57:31.754845 17753 leader_election.cc:304] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:31.755407 18220 tablet_bootstrap.cc:492] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:31.755288 18224 consensus_queue.cc:260] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.756523 18536 raft_consensus.cc:2804] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:31.758731 17972 raft_consensus.cc:2468] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:31.759891 17753 leader_election.cc:304] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:31.764673 18224 ts_tablet_manager.cc:1434] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260924 19:57:31.768790 18220 tablet_bootstrap.cc:654] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.773664 18224 tablet_bootstrap.cc:492] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:31.770761 18540 raft_consensus.cc:2804] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:31.779186 18540 raft_consensus.cc:697] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:31.783936 18540 consensus_queue.cc:237] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.790135 18224 tablet_bootstrap.cc:654] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.800953 18498 leader_election.cc:290] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.802649 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2440e1cade4327a63d6234e3adcd64" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:31.803429 17823 raft_consensus.cc:3060] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.803231 18508 consensus_queue.cc:1048] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:31.804963 18508 consensus_queue.cc:1048] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:31.806044 18220 tablet_bootstrap.cc:492] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:31.806501 18220 ts_tablet_manager.cc:1403] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.051s user 0.014s sys 0.023s
I20260924 19:57:31.809913 18220 raft_consensus.cc:359] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.819088 18290 consensus_queue.cc:1048] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:31.820636 18231 ts_tablet_manager.cc:1929] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:31.821089 18231 log.cc:1199] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/a384c9a0bb65429ebb97fe0807c78773
I20260924 19:57:31.821964 18231 ts_tablet_manager.cc:1950] T a384c9a0bb65429ebb97fe0807c78773 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:31.826308 18220 raft_consensus.cc:385] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.826674 18220 raft_consensus.cc:740] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.827426 18220 consensus_queue.cc:260] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.829629 18220 ts_tablet_manager.cc:1434] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.023s user 0.003s sys 0.004s
I20260924 19:57:31.830605 18220 tablet_bootstrap.cc:492] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:31.831907 18540 raft_consensus.cc:493] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:31.832434 18540 raft_consensus.cc:515] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.835462 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "954178b8cc1e434085337f66b0e85672" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:31.836668 17753 leader_election.cc:343] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:31.837390 18290 raft_consensus.cc:493] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:31.837877 18290 raft_consensus.cc:515] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.840147 18290 leader_election.cc:290] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.841418 18540 leader_election.cc:290] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.841666 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c884f12860164f12bbcd0d2f87067013" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:31.842221 17967 raft_consensus.cc:2468] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:31.843263 17753 leader_election.cc:304] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:31.844130 18540 raft_consensus.cc:2804] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:31.844501 18540 raft_consensus.cc:493] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:31.844856 18540 raft_consensus.cc:3060] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.850036 17606 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 5f994e0401df4fc19bc5abb10f08bc6e (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]). Expected version 1 got 0
I20260924 19:57:31.866008 18508 raft_consensus.cc:493] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:31.866487 18508 raft_consensus.cc:515] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.847082 17823 raft_consensus.cc:2468] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:31.869750 18508 leader_election.cc:290] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.883805 17798 tablet_service.cc:1511] Processing CreateTablet for tablet e68378312c334fc28dafb71bf6386be8 (DEFAULT_TABLE table=default.Table_8be3db507b2d42da90969bacc361c127 [id=cd5ff543cb1f48f58d5867aef9706963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.884902 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68378312c334fc28dafb71bf6386be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:31.887542 18224 tablet_bootstrap.cc:492] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:31.888016 18224 ts_tablet_manager.cc:1403] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.123s user 0.014s sys 0.006s
I20260924 19:57:31.893682 18220 tablet_bootstrap.cc:654] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.895035 18508 raft_consensus.cc:493] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:31.895537 18508 raft_consensus.cc:515] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.896519 18540 raft_consensus.cc:515] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.898526 18508 leader_election.cc:290] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.899739 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c884f12860164f12bbcd0d2f87067013" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:31.900460 17972 raft_consensus.cc:3060] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:31.874908 17608 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 144) took 1096 ms. Trace:
I20260924 19:57:31.879243 17612 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 145) took 1100 ms. Trace:
I20260924 19:57:31.907279 18536 raft_consensus.cc:697] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:31.908104 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f38188eccc344246b72a8755ec5201ed" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:31.908836 17821 raft_consensus.cc:2468] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:31.908326 18536 consensus_queue.cc:237] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.912935 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "201ea0020cb3430eac85e80b8c06fb07" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:31.913550 17821 raft_consensus.cc:2468] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:31.920102 18348 ts_tablet_manager.cc:1929] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:31.926069 18540 leader_election.cc:290] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.890844 18224 raft_consensus.cc:359] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.927186 18224 raft_consensus.cc:385] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:31.927498 18224 raft_consensus.cc:740] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:31.928229 18224 consensus_queue.cc:260] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.927361 18348 log.cc:1199] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/a384c9a0bb65429ebb97fe0807c78773
I20260924 19:57:31.930599 18224 ts_tablet_manager.cc:1434] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.042s user 0.007s sys 0.000s
I20260924 19:57:31.931632 18348 ts_tablet_manager.cc:1950] T a384c9a0bb65429ebb97fe0807c78773 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:31.921768 17952 tablet_service.cc:1511] Processing CreateTablet for tablet e68378312c334fc28dafb71bf6386be8 (DEFAULT_TABLE table=default.Table_8be3db507b2d42da90969bacc361c127 [id=cd5ff543cb1f48f58d5867aef9706963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.921244 17972 raft_consensus.cc:2468] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:31.936828 17753 leader_election.cc:304] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:31.937597 18540 raft_consensus.cc:2804] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:31.938410 18540 raft_consensus.cc:697] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:31.939402 18540 consensus_queue.cc:237] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.941365 18413 consensus_queue.cc:1048] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:31.942080 18426 consensus_queue.cc:1048] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:31.947343 18224 tablet_bootstrap.cc:492] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:31.979594 18540 raft_consensus.cc:493] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:31.980398 17951 tablet_service.cc:1511] Processing CreateTablet for tablet 6b7f2dd8ba2a4cb190e9ff92a542bacb (DEFAULT_TABLE table=default.Table_11ca95d108694c9498d4de67c9ff8808 [id=bec51c14454046329b3eb7016389f235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.980197 18540 raft_consensus.cc:515] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:31.982064 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68378312c334fc28dafb71bf6386be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:31.982846 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b7f2dd8ba2a4cb190e9ff92a542bacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:31.983801 18540 leader_election.cc:290] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:31.984216 18224 tablet_bootstrap.cc:654] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:31.988427 18231 tablet_replica.cc:333] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:31.989166 18231 raft_consensus.cc:2243] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:31.989581 18231 raft_consensus.cc:2272] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:31.903931 17608 rpcz_store.cc:276] 0924 19:57:30.778780 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.212524 (+433744us) service_pool.cc:224] Handling call
0924 19:57:31.252683 (+ 40159us) spinlock_profiling.cc:243] Waited 38.7 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552207166 00007fa54fdb3034 00007fa54fdb2fc2 00007fa54fdb2ef1 00007fa54fdb2e6c 00007fa54fdb2122 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:31.256021 (+ 3338us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:31.258321 (+ 2300us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:31.874169 (+615848us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:31.874524 (+ 355us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:31.874843 (+ 319us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:31.741067 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:31.741507 (+ 440us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:31.741534 (+ 27us) write_op.cc:435] Acquired schema lock
0924 19:57:31.741551 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:31.741866 (+ 315us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:31.741912 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:31.741939 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:31.742160 (+ 221us) tablet.cc:701] Row locks acquired
0924 19:57:31.742175 (+ 15us) write_op.cc:260] PREPARE: finished
0924 19:57:31.742299 (+ 124us) write_op.cc:270] Start()
0924 19:57:31.742389 (+ 90us) write_op.cc:276] Timestamp: P: 1790279851742234 usec, L: 0
0924 19:57:31.742407 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:31.837573 (+ 95166us) log.cc:844] Serialized 433 byte log entry
0924 19:57:31.838176 (+ 603us) spinlock_profiling.cc:243] Waited 94.6 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f7aa1 00007fa55038a5e8 00007fa55081a741 00007fa550819f18 00007fa55081ebb1 00007fa55081eb69 00007fa55081eaf9 00007fa55081eac1 00007fa55081dced 00000000005bd4e4 00000000005bd319
0924 19:57:31.855037 (+ 16861us) spinlock_profiling.cc:243] Waited 8.43 ms on lock 0x7b58000401d0. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5502e896c 00007fa5502ef283 00007fa5502ef209 00007fa5502ef199 00007fa5502ef161 00007fa5502ee32d 00000000005bd4e4 00000000005bd319 00007fa54922c3e5 00007fa54922e751
0924 19:57:31.858901 (+ 3864us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b5400030d98. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f7aa1 00007fa55034af75 00007fa55034b0c2 00007fa5503077bf 00007fa5502e83da 00007fa5502ef283 00007fa5502ef209 00007fa5502ef199 00007fa5502ef161 00007fa5502ee32d 00000000005bd4e4
0924 19:57:31.872325 (+ 13424us) op_driver.cc:464] REPLICATION: finished
0924 19:57:31.872545 (+ 220us) write_op.cc:301] APPLY: starting
0924 19:57:31.872602 (+ 57us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:31.872706 (+ 104us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:31.872725 (+ 19us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:31.873143 (+ 418us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:31.873284 (+ 141us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:31.873348 (+ 64us) write_op.cc:312] APPLY: finished
0924 19:57:31.873537 (+ 189us) log.cc:844] Serialized 29 byte log entry
0924 19:57:31.873682 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:31.873751 (+ 69us) write_op.cc:454] Released schema lock
0924 19:57:31.873793 (+ 42us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":88901888,"child_traces":[["op",{"apply.queue_time_us":79,"apply.run_cpu_time_us":1445,"apply.run_wall_time_us":1444,"mutex_wait_us":5131,"num_ops":2,"prepare.queue_time_us":481180,"prepare.run_cpu_time_us":3733,"prepare.run_wall_time_us":102523,"raft.queue_time_us":9091,"raft.run_cpu_time_us":3138,"raft.run_wall_time_us":23963,"replication_time_us":129816,"spinlock_wait_cycles":265982976}]]}
I20260924 19:57:31.994488 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_f40037fb3c91495e83bfe25056d5012f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:31.997783 18224 tablet_bootstrap.cc:492] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:31.998188 18224 ts_tablet_manager.cc:1403] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.000s
I20260924 19:57:31.905560 17612 rpcz_store.cc:276] 0924 19:57:30.779036 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.247507 (+468471us) service_pool.cc:224] Handling call
0924 19:57:31.260514 (+ 13007us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552207166 00007fa54fdb3034 00007fa54fdb2fc2 00007fa54fdb2ef1 00007fa54fdb2e6c 00007fa54fdb2122 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:31.263108 (+ 2594us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:31.544143 (+281035us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:31.878658 (+334515us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:31.878923 (+ 265us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:31.879186 (+ 263us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:31.843694 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:31.844099 (+ 405us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:31.844123 (+ 24us) write_op.cc:435] Acquired schema lock
0924 19:57:31.844137 (+ 14us) tablet.cc:662] Decoding operations
0924 19:57:31.844394 (+ 257us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:31.844429 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:31.844451 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:31.844637 (+ 186us) tablet.cc:701] Row locks acquired
0924 19:57:31.844651 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:31.844757 (+ 106us) write_op.cc:270] Start()
0924 19:57:31.844857 (+ 100us) write_op.cc:276] Timestamp: P: 1790279851844684 usec, L: 0
0924 19:57:31.844873 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:31.856444 (+ 11571us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b5400030d98. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa551135964 00007fa550349e69 00007fa550305b51 00007fa55030582a 00007fa55038a125 00007fa55038a5c9 00007fa55081a741 00007fa550819f18 00007fa55081ebb1 00007fa55081eb69 00007fa55081eaf9
0924 19:57:31.856767 (+ 323us) log.cc:844] Serialized 433 byte log entry
0924 19:57:31.875682 (+ 18915us) op_driver.cc:464] REPLICATION: finished
0924 19:57:31.877154 (+ 1472us) write_op.cc:301] APPLY: starting
0924 19:57:31.877210 (+ 56us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:31.877314 (+ 104us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:31.877331 (+ 17us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:31.877666 (+ 335us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:31.877798 (+ 132us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:31.877840 (+ 42us) write_op.cc:312] APPLY: finished
0924 19:57:31.877999 (+ 159us) log.cc:844] Serialized 29 byte log entry
0924 19:57:31.878112 (+ 113us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:31.878173 (+ 61us) write_op.cc:454] Released schema lock
0924 19:57:31.878203 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:31.878396 (+ 193us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:31.878418 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":28083968,"child_traces":[["op",{"apply.queue_time_us":1331,"apply.run_cpu_time_us":1218,"apply.run_wall_time_us":1214,"num_ops":2,"prepare.queue_time_us":298249,"prepare.run_cpu_time_us":3322,"prepare.run_wall_time_us":22132,"replication_time_us":30719,"spinlock_wait_cycles":23255296}]]}
I20260924 19:57:32.000236 17612 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_0bb01b569e3c41a4a53d6eb7817aaade" } modify_external_catalogs: true
I20260924 19:57:32.000720 17612 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_0bb01b569e3c41a4a53d6eb7817aaade" } modify_external_catalogs: true
I20260924 19:57:32.000730 18224 raft_consensus.cc:359] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.001430 18224 raft_consensus.cc:385] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:32.001701 18224 raft_consensus.cc:740] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:32.007970 18224 consensus_queue.cc:260] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:32.010285 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f40037fb3c91495e83bfe25056d5012f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:32.011829 18348 tablet_replica.cc:333] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:32.012629 18348 raft_consensus.cc:2243] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:32.013057 18348 raft_consensus.cc:2272] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:32.015851 18224 ts_tablet_manager.cc:1434] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.017s user 0.008s sys 0.000s
I20260924 19:57:32.024341 18224 tablet_bootstrap.cc:492] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:32.035244 18231 ts_tablet_manager.cc:1916] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:32.040135 18224 tablet_bootstrap.cc:654] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:32.061578 17798 tablet_service.cc:1511] Processing CreateTablet for tablet 6b7f2dd8ba2a4cb190e9ff92a542bacb (DEFAULT_TABLE table=default.Table_11ca95d108694c9498d4de67c9ff8808 [id=bec51c14454046329b3eb7016389f235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.062649 17800 tablet_service.cc:1511] Processing CreateTablet for tablet fd6e656a653a43d1843a91fee813237d (DEFAULT_TABLE table=default.Table_e3179b11ea114ce99779ed0b2ac46732 [id=8c39c8f34038451d840db3c909d76f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.063865 17799 tablet_service.cc:1511] Processing CreateTablet for tablet 41bdb846ca19425b882434df0167479e (DEFAULT_TABLE table=default.Table_0bb01b569e3c41a4a53d6eb7817aaade [id=ad9f72129d8343e0932f8035a6554aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.063802 17797 tablet_service.cc:1511] Processing CreateTablet for tablet 2cf62a73c49245198d65011875148dc3 (DEFAULT_TABLE table=default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.064296 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 9c861e93ad9f40ba9ad010021e43bc2b (DEFAULT_TABLE table=default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.062664 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b7f2dd8ba2a4cb190e9ff92a542bacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.065081 17802 tablet_service.cc:1511] Processing CreateTablet for tablet c1684aaf9d984649bff8c4b0b12044b9 (DEFAULT_TABLE table=default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.063024 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet a384c9a0bb65429ebb97fe0807c78773 (table default.Table_1a379ab4fc834d69865de7106e463200 [id=3ee2ed8b290747128a29f7c107dbfe82]) successfully deleted
I20260924 19:57:32.065505 17796 tablet_service.cc:1511] Processing CreateTablet for tablet a765abd702644f079bddffb3fc4238ef (DEFAULT_TABLE table=default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.066551 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41bdb846ca19425b882434df0167479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.067500 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6e656a653a43d1843a91fee813237d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.068125 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c861e93ad9f40ba9ad010021e43bc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.069073 17606 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 97a1b8ccd35a43cea7a71d705fad1494 (table default.Table_850e5158d78d4850b87445dac75ef7a9 [id=f768ce157a7b407d9f1e55a5b9d0720a]). Expected version 1 got 0
I20260924 19:57:32.071405 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1684aaf9d984649bff8c4b0b12044b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.070658 17606 catalog_manager.cc:5838] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:32.075918 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a765abd702644f079bddffb3fc4238ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.084004 17797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf62a73c49245198d65011875148dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.091689 18348 ts_tablet_manager.cc:1916] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:32.097455 17795 tablet_service.cc:1511] Processing CreateTablet for tablet 872bc32b9e384c828a0699079b27530d (DEFAULT_TABLE table=default.Table_3bf03540f35b468dacc7a86dd2d3d2a7 [id=eaf422968e464f1986f89b6944a6adbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.099596 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872bc32b9e384c828a0699079b27530d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.108503 18231 ts_tablet_manager.cc:1929] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:32.108924 18231 log.cc:1199] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/f38188eccc344246b72a8755ec5201ed
I20260924 19:57:32.109552 18231 ts_tablet_manager.cc:1950] T f38188eccc344246b72a8755ec5201ed P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:32.112010 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2e4519aed7642678db3a518a89778de" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:31.924870 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 393933us
W20260924 19:57:31.936586 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 361203us
I20260924 19:57:31.951753 18094 tablet_service.cc:1511] Processing CreateTablet for tablet 910d61b8c3574302b5a328187226571e (DEFAULT_TABLE table=default.Table_2b157ac766d244e081168b817296bd4b [id=1bef6766d6a049a9a5126339c3f67842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.952862 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 0cf2df461598411f8ad9f3a51eafc6f7 (DEFAULT_TABLE table=default.Table_a10031a0fa004ac3ad48e6020477494b [id=7aab4014a5bc4823bbe845c021153fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.951754 18095 tablet_service.cc:1511] Processing CreateTablet for tablet e68378312c334fc28dafb71bf6386be8 (DEFAULT_TABLE table=default.Table_8be3db507b2d42da90969bacc361c127 [id=cd5ff543cb1f48f58d5867aef9706963]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:31.962451 18553 raft_consensus.cc:493] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:31.965689 18098 consensus_meta.cc:287] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent flushing consensus metadata: real 0.536s user 0.008s sys 0.002s
I20260924 19:57:31.969137 18114 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "201ea0020cb3430eac85e80b8c06fb07" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:32.118139 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2440e1cade4327a63d6234e3adcd64" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
W20260924 19:57:32.124276 17753 leader_election.cc:343] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:32.127413 18553 raft_consensus.cc:515] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.131088 18096 tablet_service.cc:1511] Processing CreateTablet for tablet 6b7f2dd8ba2a4cb190e9ff92a542bacb (DEFAULT_TABLE table=default.Table_11ca95d108694c9498d4de67c9ff8808 [id=bec51c14454046329b3eb7016389f235]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.133545 18540 consensus_queue.cc:1048] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:32.135408 18492 log.cc:1199] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/a384c9a0bb65429ebb97fe0807c78773
I20260924 19:57:32.136123 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet f38188eccc344246b72a8755ec5201ed (table default.Table_37482fb26ba144619bb66861dceaab3b [id=06b8b073d6ae475c8125613b89463d3c]) successfully deleted
I20260924 19:57:32.136435 18492 ts_tablet_manager.cc:1950] T a384c9a0bb65429ebb97fe0807c78773 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:32.139508 18114 raft_consensus.cc:2468] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:32.140408 18525 raft_consensus.cc:493] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:32.140856 18525 raft_consensus.cc:515] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.145375 18525 leader_election.cc:290] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:32.145840 18525 raft_consensus.cc:493] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:32.146207 18525 raft_consensus.cc:515] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.153602 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2440e1cade4327a63d6234e3adcd64" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:32.154232 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cf2df461598411f8ad9f3a51eafc6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.154505 17821 raft_consensus.cc:2393] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eaf64bfb64aa42948c9bc1411ce4ce35 in current term 1: Already voted for candidate f74e6ed351284d7fb110e44336ce643a in this term.
I20260924 19:57:32.155740 17906 leader_election.cc:304] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:32.157199 18220 tablet_bootstrap.cc:492] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:32.158121 18220 ts_tablet_manager.cc:1403] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.328s user 0.016s sys 0.036s
I20260924 19:57:32.157963 18413 raft_consensus.cc:2804] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:32.158696 18413 raft_consensus.cc:697] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:32.159543 18413 consensus_queue.cc:237] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.162670 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 41bdb846ca19425b882434df0167479e (DEFAULT_TABLE table=default.Table_0bb01b569e3c41a4a53d6eb7817aaade [id=ad9f72129d8343e0932f8035a6554aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.164116 18220 raft_consensus.cc:359] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.164842 18220 raft_consensus.cc:385] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:32.165078 18220 raft_consensus.cc:740] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
W20260924 19:57:32.165416 17602 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 349c6e9413bb41268d0c41d549203e2a (table default.Table_41c56bb86db34f16aba6e94147cde1d2 [id=b1d9f8b4039b4d19bfc493355e0d07cc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 349c6e9413bb41268d0c41d549203e2a consensus state has no leader
I20260924 19:57:32.173363 18220 consensus_queue.cc:260] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.170214 18116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f38188eccc344246b72a8755ec5201ed" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:32.177551 18116 raft_consensus.cc:2468] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:32.178941 18220 ts_tablet_manager.cc:1434] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20260924 19:57:32.144543 18115 raft_consensus.cc:3060] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:32.182780 18220 tablet_bootstrap.cc:492] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:32.187582 17906 leader_election.cc:304] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
W20260924 19:57:32.188581 17906 raft_consensus.cc:2686] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260924 19:57:32.189145 17906 leader_election.cc:304] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:32.190080 18426 raft_consensus.cc:2804] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:32.190425 18426 raft_consensus.cc:493] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:32.190760 18426 raft_consensus.cc:3060] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:32.196749 18223 tablet_bootstrap.cc:492] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:32.197412 18223 ts_tablet_manager.cc:1403] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.637s user 0.013s sys 0.007s
I20260924 19:57:32.211472 18224 tablet_bootstrap.cc:492] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:32.211877 18224 ts_tablet_manager.cc:1403] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.188s user 0.021s sys 0.000s
I20260924 19:57:32.213614 17612 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 41bdb846ca19425b882434df0167479e
I20260924 19:57:32.215538 17612 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 147) took 1095 ms. Trace:
I20260924 19:57:32.216166 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet 41bdb846ca19425b882434df0167479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:31 UTC) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:32.218902 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 41bdb846ca19425b882434df0167479e because the tablet was not found. No further retry: Not found: Tablet not found: 41bdb846ca19425b882434df0167479e
I20260924 19:57:32.231395 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet a384c9a0bb65429ebb97fe0807c78773 (table default.Table_1a379ab4fc834d69865de7106e463200 [id=3ee2ed8b290747128a29f7c107dbfe82]) successfully deleted
I20260924 19:57:32.215968 17612 rpcz_store.cc:276] 0924 19:57:31.120467 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.999972 (+879505us) service_pool.cc:224] Handling call
0924 19:57:32.001109 (+ 1137us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:32.002913 (+ 1804us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:32.003023 (+ 110us) catalog_manager.cc:2863] Locking tablets
0924 19:57:32.003642 (+ 619us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:32.212712 (+209070us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:32.213090 (+ 378us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:32.213205 (+ 115us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:32.215423 (+ 2218us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:32.005399 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:32.005733 (+ 334us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:32.005756 (+ 23us) write_op.cc:435] Acquired schema lock
0924 19:57:32.005769 (+ 13us) tablet.cc:662] Decoding operations
0924 19:57:32.006053 (+ 284us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:32.006091 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:32.006113 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:32.006348 (+ 235us) tablet.cc:701] Row locks acquired
0924 19:57:32.006361 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:57:32.006466 (+ 105us) write_op.cc:270] Start()
0924 19:57:32.006544 (+ 78us) write_op.cc:276] Timestamp: P: 1790279852006377 usec, L: 0
0924 19:57:32.006559 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:57:32.007003 (+ 444us) log.cc:844] Serialized 728 byte log entry
0924 19:57:32.092910 (+ 85907us) op_driver.cc:464] REPLICATION: finished
0924 19:57:32.210391 (+117481us) write_op.cc:301] APPLY: starting
0924 19:57:32.210479 (+ 88us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:32.210614 (+ 135us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:32.210740 (+ 126us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:32.211100 (+ 360us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:32.211302 (+ 202us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:32.211355 (+ 53us) write_op.cc:312] APPLY: finished
0924 19:57:32.211572 (+ 217us) log.cc:844] Serialized 29 byte log entry
0924 19:57:32.212298 (+ 726us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:32.212388 (+ 90us) write_op.cc:454] Released schema lock
0924 19:57:32.212425 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:32.214030 (+ 1605us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:32.214057 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":172,"child_traces":[["op",{"apply.queue_time_us":117220,"apply.run_cpu_time_us":1762,"apply.run_wall_time_us":3541,"num_ops":2,"prepare.queue_time_us":99,"prepare.run_cpu_time_us":1896,"prepare.run_wall_time_us":3936,"raft.queue_time_us":82,"raft.run_cpu_time_us":647,"raft.run_wall_time_us":647,"replication_time_us":86114}]]}
I20260924 19:57:32.234647 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_202c25f88e1a485c9ca6677d9cf53e75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:32.242709 18093 tablet_service.cc:1511] Processing CreateTablet for tablet c1684aaf9d984649bff8c4b0b12044b9 (DEFAULT_TABLE table=default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:57:32.247330 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_202c25f88e1a485c9ca6677d9cf53e75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:32.247500 18116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef3f82de70bb4328a15a044fe0e500bf" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:32.248047 18116 raft_consensus.cc:3060] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:32.248575 18426 raft_consensus.cc:515] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.251564 18114 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "954178b8cc1e434085337f66b0e85672" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:32.257175 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2e4519aed7642678db3a518a89778de" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:32.240624 18117 consensus_meta.cc:287] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent flushing consensus metadata: real 1.063s user 0.008s sys 0.000s
I20260924 19:57:32.253373 18220 tablet_bootstrap.cc:654] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:32.235333 18553 leader_election.cc:290] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:32.267844 18107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c884f12860164f12bbcd0d2f87067013" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:32.268468 18107 raft_consensus.cc:3060] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:32.268592 17952 tablet_service.cc:1511] Processing CreateTablet for tablet 41bdb846ca19425b882434df0167479e (DEFAULT_TABLE table=default.Table_0bb01b569e3c41a4a53d6eb7817aaade [id=ad9f72129d8343e0932f8035a6554aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.269556 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41bdb846ca19425b882434df0167479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.273636 18105 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2e4519aed7642678db3a518a89778de" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:32.274343 18105 raft_consensus.cc:2468] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:32.274936 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2440e1cade4327a63d6234e3adcd64" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:32.276582 18109 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c884f12860164f12bbcd0d2f87067013" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:32.277192 17946 tablet_service.cc:1511] Processing CreateTablet for tablet c1684aaf9d984649bff8c4b0b12044b9 (DEFAULT_TABLE table=default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.267256 18553 raft_consensus.cc:493] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:32.278491 17946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1684aaf9d984649bff8c4b0b12044b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.266634 18117 raft_consensus.cc:2468] T 7c7b8422753a4746bde71fcb25409949 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:32.280424 18092 tablet_service.cc:1511] Processing CreateTablet for tablet fd6e656a653a43d1843a91fee813237d (DEFAULT_TABLE table=default.Table_e3179b11ea114ce99779ed0b2ac46732 [id=8c39c8f34038451d840db3c909d76f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.284417 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c2440e1cade4327a63d6234e3adcd64" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:32.285809 18553 raft_consensus.cc:515] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.287904 18553 leader_election.cc:290] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:32.290508 18115 raft_consensus.cc:2468] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:32.282773 18223 raft_consensus.cc:359] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.293530 18223 raft_consensus.cc:385] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:32.293859 18223 raft_consensus.cc:740] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:32.292698 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef3f82de70bb4328a15a044fe0e500bf" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:32.303035 17951 tablet_service.cc:1511] Processing CreateTablet for tablet fd6e656a653a43d1843a91fee813237d (DEFAULT_TABLE table=default.Table_e3179b11ea114ce99779ed0b2ac46732 [id=8c39c8f34038451d840db3c909d76f3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.304330 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6e656a653a43d1843a91fee813237d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.307590 17612 master_service.cc:960] ReplaceTablet: received request to replace tablet 9c2440e1cade4327a63d6234e3adcd64 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:32.267601 18426 leader_election.cc:290] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:32.277802 18525 leader_election.cc:290] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:32.316988 17949 tablet_service.cc:1511] Processing CreateTablet for tablet 9c861e93ad9f40ba9ad010021e43bc2b (DEFAULT_TABLE table=default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.317305 17948 tablet_service.cc:1511] Processing CreateTablet for tablet 2cf62a73c49245198d65011875148dc3 (DEFAULT_TABLE table=default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.317560 17947 tablet_service.cc:1511] Processing CreateTablet for tablet a765abd702644f079bddffb3fc4238ef (DEFAULT_TABLE table=default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.318152 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c861e93ad9f40ba9ad010021e43bc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.320106 17950 tablet_service.cc:1511] Processing CreateTablet for tablet 872bc32b9e384c828a0699079b27530d (DEFAULT_TABLE table=default.Table_3bf03540f35b468dacc7a86dd2d3d2a7 [id=eaf422968e464f1986f89b6944a6adbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.321419 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872bc32b9e384c828a0699079b27530d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:32.265018 18320 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:57:32.324446 18320 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:57:32.325394 18320 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260924 19:57:32.214680 18224 raft_consensus.cc:359] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.348760 18224 raft_consensus.cc:385] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:32.350258 17608 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 146) took 1511 ms. Trace:
I20260924 19:57:32.349085 18224 raft_consensus.cc:740] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:32.353047 18224 consensus_queue.cc:260] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.351584 17608 rpcz_store.cc:276] 0924 19:57:30.838698 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.994107 (+1155409us) service_pool.cc:224] Handling call
0924 19:57:32.010714 (+ 16607us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:32.012285 (+ 1571us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:32.347967 (+335682us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:32.349031 (+ 1064us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:32.349669 (+ 638us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:32.013661 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:32.014019 (+ 358us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:32.014038 (+ 19us) write_op.cc:435] Acquired schema lock
0924 19:57:32.014048 (+ 10us) tablet.cc:662] Decoding operations
0924 19:57:32.014303 (+ 255us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:32.014336 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:32.014357 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:32.014537 (+ 180us) tablet.cc:701] Row locks acquired
0924 19:57:32.014550 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:57:32.014653 (+ 103us) write_op.cc:270] Start()
0924 19:57:32.014731 (+ 78us) write_op.cc:276] Timestamp: P: 1790279852014562 usec, L: 0
0924 19:57:32.014746 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:57:32.015206 (+ 460us) log.cc:844] Serialized 433 byte log entry
0924 19:57:32.106957 (+ 91751us) op_driver.cc:464] REPLICATION: finished
0924 19:57:32.214351 (+107394us) write_op.cc:301] APPLY: starting
0924 19:57:32.335254 (+120903us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:32.335389 (+ 135us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:32.335484 (+ 95us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:32.335900 (+ 416us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:32.336217 (+ 317us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:32.336259 (+ 42us) write_op.cc:312] APPLY: finished
0924 19:57:32.340024 (+ 3765us) log.cc:844] Serialized 29 byte log entry
0924 19:57:32.343511 (+ 3487us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:32.343590 (+ 79us) write_op.cc:454] Released schema lock
0924 19:57:32.343626 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:32.343892 (+ 266us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:32.343917 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":107238,"apply.run_cpu_time_us":1901,"apply.run_wall_time_us":129490,"num_ops":2,"prepare.queue_time_us":94,"prepare.run_cpu_time_us":1737,"prepare.run_wall_time_us":2636,"replication_time_us":92100,"thread_start_us":107288,"threads_started":1}]]}
I20260924 19:57:32.358718 17608 master_service.cc:960] ReplaceTablet: received request to replace tablet 32466931871c4c5d8e9f95a11ffe26be from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:32.387754 17612 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 9c2440e1cade4327a63d6234e3adcd64
I20260924 19:57:32.390419 17948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf62a73c49245198d65011875148dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.392809 17612 catalog_manager.cc:6934] ReplaceTablet: tablet 9c2440e1cade4327a63d6234e3adcd64 deleted and replaced by tablet 07f86f24d9d14d05898e1c0f69428850
I20260924 19:57:32.393905 17612 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_c8e27940040c495780f92f92431541cb" } modify_external_catalogs: true
I20260924 19:57:32.394479 17612 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_c8e27940040c495780f92f92431541cb" } modify_external_catalogs: true
I20260924 19:57:32.395781 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a765abd702644f079bddffb3fc4238ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.399581 18540 raft_consensus.cc:493] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:32.400053 18540 raft_consensus.cc:515] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.401968 18540 leader_election.cc:290] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:32.398228 17607 catalog_manager.cc:5838] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:32.403798 17607 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 5f994e0401df4fc19bc5abb10f08bc6e (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]). Expected version 1 got 0
I20260924 19:57:32.404855 17607 catalog_manager.cc:5838] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:32.405995 17607 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 97a1b8ccd35a43cea7a71d705fad1494 (table default.Table_850e5158d78d4850b87445dac75ef7a9 [id=f768ce157a7b407d9f1e55a5b9d0720a]). Expected version 1 got 0
I20260924 19:57:32.414983 18539 raft_consensus.cc:493] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:32.415491 18539 raft_consensus.cc:515] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.417322 18539 leader_election.cc:290] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:32.417732 18539 raft_consensus.cc:493] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:32.418085 18539 raft_consensus.cc:515] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:32.424265 17753 leader_election.cc:343] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:32.422848 18091 tablet_service.cc:1511] Processing CreateTablet for tablet 9c861e93ad9f40ba9ad010021e43bc2b (DEFAULT_TABLE table=default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.425069 17753 leader_election.cc:304] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:32.425922 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68378312c334fc28dafb71bf6386be8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.294502 18223 consensus_queue.cc:260] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.431051 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "201ea0020cb3430eac85e80b8c06fb07" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:32.431614 17821 raft_consensus.cc:3060] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:32.431919 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b7f2dd8ba2a4cb190e9ff92a542bacb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.433050 17945 tablet_service.cc:1558] Processing DeleteTablet for tablet 41bdb846ca19425b882434df0167479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:31 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:32.434713 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "151247d6b5254cd884c58db9311ac04a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:32.434909 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a524bad1c4040e58ad84fcf5a4fc47d" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:32.426497 18540 raft_consensus.cc:2749] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:32.291967 18117 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.2.130:37897 (request call id 37) took 1160 ms. Trace:
I20260924 19:57:32.437901 17972 raft_consensus.cc:2468] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
W20260924 19:57:32.438309 17753 leader_election.cc:343] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:32.406725 17607 catalog_manager.cc:5838] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:32.441592 18094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 910d61b8c3574302b5a328187226571e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:32.442085 17753 consensus_peers.cc:597] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 085b8088134d47478f0ce06a6a9f8572. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:32.440966 17607 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 349c6e9413bb41268d0c41d549203e2a (table default.Table_41c56bb86db34f16aba6e94147cde1d2 [id=b1d9f8b4039b4d19bfc493355e0d07cc]). Expected version 1 got 0
I20260924 19:57:32.442935 18539 leader_election.cc:290] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:32.447489 17609 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_7c382a08aec546da8c1aa720956036c7 [id=31e1c3788d264823a12ecd9755ec2027] alter complete (version 4)
W20260924 19:57:32.449100 17609 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 22) took 2931 ms (client timeout 1000 ms). Trace:
I20260924 19:57:32.446408 17753 leader_election.cc:304] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:32.450907 18539 raft_consensus.cc:2804] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:32.451344 18539 raft_consensus.cc:493] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 19:57:32.451833 17613 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 077f70bcee7e412299da12ddc30a63ff consensus state has no leader
I20260924 19:57:32.452368 17613 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 136) took 2179 ms. Trace:
I20260924 19:57:32.452654 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd434e38c9364f55bd18383149015028" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:32.453154 18539 raft_consensus.cc:3060] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:32.457116 18426 consensus_queue.cc:1048] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:32.458050 18540 consensus_queue.cc:1048] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:32.459370 17801 tablet_service.cc:1511] Processing CreateTablet for tablet c0feecd2fdb4409f86f6a5fbfbab3537 (DEFAULT_TABLE table=default.Table_5e50f3dbbb674d658a5afac0ebcedaa6 [id=545488f05d324b748e42dae4a665f0d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.460331 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0feecd2fdb4409f86f6a5fbfbab3537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.460635 17821 raft_consensus.cc:2468] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
W20260924 19:57:32.458984 17753 leader_election.cc:343] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:32.462448 17610 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 33) took 2431 ms (client timeout 1000 ms). Trace:
I20260924 19:57:32.463534 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c2440e1cade4327a63d6234e3adcd64 with delete_type TABLET_DATA_DELETED (replaced by tablet 07f86f24d9d14d05898e1c0f69428850) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:32.465350 18231 tablet_replica.cc:333] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:32.466485 18231 raft_consensus.cc:2243] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:32.467052 18231 raft_consensus.cc:2272] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:32.471634 18539 raft_consensus.cc:515] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.474249 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "151247d6b5254cd884c58db9311ac04a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:32.474879 17972 raft_consensus.cc:3060] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:32.477766 17796 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a19b5812dc54589a5cbf48199951600 with delete_type TABLET_DATA_DELETED (replaced by tablet f38188eccc344246b72a8755ec5201ed) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:32.479784 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 0c08f10d2a364cb8a0a7ea40654c3fe4 (DEFAULT_TABLE table=default.Table_2bb6cac7492d4c5ca461d126788c9722 [id=f1f88aa92d884988b4bdf211754de8de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.480973 17799 tablet_service.cc:1511] Processing CreateTablet for tablet dbb82914df65416aaf47db8e507afb40 (DEFAULT_TABLE table=default.Table_79bc1edaa2024011a001b435d63329f4 [id=56af0067ee3548a29ee663929dfc9140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.478873 17797 tablet_service.cc:1511] Processing CreateTablet for tablet d970cc92a1134a98a613d198a319cb01 (DEFAULT_TABLE table=default.Table_a50429edeaf645668bc84244412bb359 [id=1853f611de514a55afd7a68d742503b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.481649 18224 ts_tablet_manager.cc:1434] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.270s user 0.007s sys 0.000s
I20260924 19:57:32.482705 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c08f10d2a364cb8a0a7ea40654c3fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.485587 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb82914df65416aaf47db8e507afb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.487622 17797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d970cc92a1134a98a613d198a319cb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.488909 17796 tablet_service.cc:1511] Processing CreateTablet for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb (DEFAULT_TABLE table=default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.489889 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.493587 17612 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 151247d6b5254cd884c58db9311ac04a
W20260924 19:57:32.467042 17611 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 31) took 1956 ms (client timeout 1000 ms). Trace:
I20260924 19:57:32.498226 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 9b04059219084f608e8cb5fa4d0e0d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:32.449404 17609 rpcz_store.cc:269] 0924 19:57:29.518145 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:29.609732 (+ 91587us) service_pool.cc:224] Handling call
0924 19:57:32.449027 (+2839295us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:30.469185 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:30.469702 (+ 517us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:30.469738 (+ 36us) write_op.cc:435] Acquired schema lock
0924 19:57:30.469759 (+ 21us) tablet.cc:662] Decoding operations
0924 19:57:30.470016 (+ 257us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:30.470050 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:30.470070 (+ 20us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:30.470227 (+ 157us) tablet.cc:701] Row locks acquired
0924 19:57:30.470241 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:30.470349 (+ 108us) write_op.cc:270] Start()
0924 19:57:30.470433 (+ 84us) write_op.cc:276] Timestamp: P: 1790279850469858 usec, L: 0
0924 19:57:30.470447 (+ 14us) op_driver.cc:348] REPLICATION: starting
0924 19:57:30.577416 (+106969us) log.cc:844] Serialized 356 byte log entry
0924 19:57:30.577997 (+ 581us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f7aa1 00007fa55038a5e8 00007fa55081a741 00007fa550819f18 00007fa55081ebb1 00007fa55081eb69 00007fa55081eaf9 00007fa55081eac1 00007fa55081dced 00000000005bd4e4 00000000005bd319
0924 19:57:30.615792 (+ 37795us) op_driver.cc:464] REPLICATION: finished
0924 19:57:30.619817 (+ 4025us) write_op.cc:301] APPLY: starting
0924 19:57:30.619889 (+ 72us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:30.620001 (+ 112us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:30.620021 (+ 20us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:30.620170 (+ 149us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:30.620295 (+ 125us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:30.620338 (+ 43us) write_op.cc:312] APPLY: finished
0924 19:57:30.621610 (+ 1272us) log.cc:844] Serialized 23 byte log entry
0924 19:57:30.621758 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:30.621817 (+ 59us) write_op.cc:454] Released schema lock
0924 19:57:30.621847 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:30.622705 (+ 858us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:30.622730 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":3459,"apply.run_cpu_time_us":1204,"apply.run_wall_time_us":3253,"num_ops":1,"prepare.queue_time_us":3655,"prepare.run_cpu_time_us":3466,"prepare.run_wall_time_us":109017,"raft.queue_time_us":5178,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":16,"replication_time_us":145246,"spinlock_wait_cycles":244904704}]]}
I20260924 19:57:32.503517 18224 tablet_bootstrap.cc:492] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:32.503700 17615 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 23) took 1977 ms (client timeout 1000 ms). Trace:
W20260924 19:57:32.504104 17615 rpcz_store.cc:269] 0924 19:57:30.525095 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:30.563441 (+ 38346us) service_pool.cc:224] Handling call
0924 19:57:30.587923 (+ 24482us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:32.502817 (+1914894us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":9,"spinlock_wait_cycles":51141504}
I20260924 19:57:32.452627 17613 rpcz_store.cc:276] 0924 19:57:30.273242 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:30.273420 (+ 178us) service_pool.cc:224] Handling call
0924 19:57:30.274746 (+ 1326us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:30.275868 (+ 1122us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:30.276836 (+ 968us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:30.277101 (+ 265us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:30.553625 (+276524us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:32.452292 (+1898667us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:30.279456 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:30.280045 (+ 589us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:30.280085 (+ 40us) write_op.cc:435] Acquired schema lock
0924 19:57:30.280107 (+ 22us) tablet.cc:662] Decoding operations
0924 19:57:30.280469 (+ 362us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:30.280522 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:30.280549 (+ 27us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:30.280765 (+ 216us) tablet.cc:701] Row locks acquired
0924 19:57:30.280788 (+ 23us) write_op.cc:260] PREPARE: finished
0924 19:57:30.280939 (+ 151us) write_op.cc:270] Start()
0924 19:57:30.281059 (+ 120us) write_op.cc:276] Timestamp: P: 1790279850280848 usec, L: 0
0924 19:57:30.281084 (+ 25us) op_driver.cc:348] REPLICATION: starting
0924 19:57:30.281759 (+ 675us) log.cc:844] Serialized 405 byte log entry
0924 19:57:30.545307 (+263548us) op_driver.cc:464] REPLICATION: finished
0924 19:57:30.551610 (+ 6303us) write_op.cc:301] APPLY: starting
0924 19:57:30.551664 (+ 54us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:30.551759 (+ 95us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:30.551838 (+ 79us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:30.552004 (+ 166us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:30.552131 (+ 127us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:30.552175 (+ 44us) write_op.cc:312] APPLY: finished
0924 19:57:30.552346 (+ 171us) log.cc:844] Serialized 23 byte log entry
0924 19:57:30.552483 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:30.552541 (+ 58us) write_op.cc:454] Released schema lock
0924 19:57:30.552572 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6168,"apply.run_cpu_time_us":1178,"apply.run_wall_time_us":2311,"mutex_wait_us":110904,"num_ops":1,"prepare.queue_time_us":158,"prepare.run_cpu_time_us":2706,"prepare.run_wall_time_us":113596,"raft.queue_time_us":113254,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":14,"replication_time_us":264131}]]}
W20260924 19:57:32.508597 17614 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 30) took 3040 ms (client timeout 1000 ms). Trace:
I20260924 19:57:32.512742 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 151247d6b5254cd884c58db9311ac04a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:32.462860 17610 rpcz_store.cc:269] 0924 19:57:30.030815 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:30.038584 (+ 7769us) service_pool.cc:224] Handling call
0924 19:57:30.124034 (+ 85450us) spinlock_profiling.cc:243] Waited 72.2 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:32.462232 (+2338198us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":166067968}
I20260924 19:57:32.520119 17945 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c2440e1cade4327a63d6234e3adcd64 with delete_type TABLET_DATA_DELETED (replaced by tablet 07f86f24d9d14d05898e1c0f69428850) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:32.523131 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 077f70bcee7e412299da12ddc30a63ff (DEFAULT_TABLE table=default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.524039 17945 tablet_service.cc:1511] Processing CreateTablet for tablet c0feecd2fdb4409f86f6a5fbfbab3537 (DEFAULT_TABLE table=default.Table_5e50f3dbbb674d658a5afac0ebcedaa6 [id=545488f05d324b748e42dae4a665f0d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.529812 18348 ts_tablet_manager.cc:1929] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:32.530172 18348 log.cc:1199] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/f38188eccc344246b72a8755ec5201ed
I20260924 19:57:32.530786 18348 ts_tablet_manager.cc:1950] T f38188eccc344246b72a8755ec5201ed P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:32.532642 18348 tablet_replica.cc:333] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:32.533288 18348 raft_consensus.cc:2243] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:32.533921 18348 pending_rounds.cc:70] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: Trying to abort 1 pending ops.
I20260924 19:57:32.534121 18348 pending_rounds.cc:77] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986274456604672 op_type: NO_OP noop_request { }
I20260924 19:57:32.538043 18348 raft_consensus.cc:2889] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:32.538458 18348 raft_consensus.cc:2272] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:32.539453 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0feecd2fdb4409f86f6a5fbfbab3537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.540777 18220 tablet_bootstrap.cc:492] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:32.541154 18220 ts_tablet_manager.cc:1403] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.359s user 0.010s sys 0.006s
W20260924 19:57:32.508944 17614 rpcz_store.cc:269] 0924 19:57:29.467824 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:29.591871 (+124047us) service_pool.cc:224] Handling call
0924 19:57:32.508427 (+2916556us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:29.951635 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:29.952004 (+ 369us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:29.952029 (+ 25us) write_op.cc:435] Acquired schema lock
0924 19:57:29.952041 (+ 12us) tablet.cc:662] Decoding operations
0924 19:57:29.952291 (+ 250us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:29.952328 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:29.952353 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:29.952509 (+ 156us) tablet.cc:701] Row locks acquired
0924 19:57:29.952523 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:29.952629 (+ 106us) write_op.cc:270] Start()
0924 19:57:29.952710 (+ 81us) write_op.cc:276] Timestamp: P: 1790279849952589 usec, L: 0
0924 19:57:29.952726 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:30.108403 (+155677us) log.cc:844] Serialized 530 byte log entry
0924 19:57:30.121083 (+ 12680us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f7aa1 00007fa55038a5e8 00007fa55081a741 00007fa550819f18 00007fa55081ebb1 00007fa55081eb69 00007fa55081eaf9 00007fa55081eac1 00007fa55081dced 00000000005bd4e4 00000000005bd319
0924 19:57:30.444076 (+322993us) op_driver.cc:464] REPLICATION: finished
0924 19:57:30.445049 (+ 973us) write_op.cc:301] APPLY: starting
0924 19:57:30.445107 (+ 58us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:30.445203 (+ 96us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:30.445221 (+ 18us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:30.445388 (+ 167us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:30.445555 (+ 167us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:30.445608 (+ 53us) write_op.cc:312] APPLY: finished
0924 19:57:30.445790 (+ 182us) log.cc:844] Serialized 23 byte log entry
0924 19:57:30.445974 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:30.446035 (+ 61us) write_op.cc:454] Released schema lock
0924 19:57:30.446065 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:30.446279 (+ 214us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:30.446306 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":790,"apply.run_cpu_time_us":1209,"apply.run_wall_time_us":1208,"num_ops":1,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":4365,"prepare.run_wall_time_us":169633,"raft.queue_time_us":7194,"raft.run_cpu_time_us":16,"raft.run_wall_time_us":16,"replication_time_us":491215,"spinlock_wait_cycles":356859648}]]}
W20260924 19:57:32.496548 17611 rpcz_store.cc:269] 0924 19:57:30.510891 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:30.548023 (+ 37132us) service_pool.cc:224] Handling call
0924 19:57:30.578900 (+ 30877us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:32.466852 (+1887952us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:31.678358 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:31.678750 (+ 392us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:31.678785 (+ 35us) write_op.cc:435] Acquired schema lock
0924 19:57:31.678799 (+ 14us) tablet.cc:662] Decoding operations
0924 19:57:31.679060 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:31.679102 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:31.679126 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:31.679313 (+ 187us) tablet.cc:701] Row locks acquired
0924 19:57:31.679328 (+ 15us) write_op.cc:260] PREPARE: finished
0924 19:57:31.679429 (+ 101us) write_op.cc:270] Start()
0924 19:57:31.679508 (+ 79us) write_op.cc:276] Timestamp: P: 1790279851679375 usec, L: 0
0924 19:57:31.679521 (+ 13us) op_driver.cc:348] REPLICATION: starting
0924 19:57:31.680004 (+ 483us) log.cc:844] Serialized 530 byte log entry
0924 19:57:31.695304 (+ 15300us) op_driver.cc:464] REPLICATION: finished
0924 19:57:31.695767 (+ 463us) write_op.cc:301] APPLY: starting
0924 19:57:31.695830 (+ 63us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:31.695921 (+ 91us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:31.696031 (+ 110us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:31.696195 (+ 164us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:31.696330 (+ 135us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:31.696381 (+ 51us) write_op.cc:312] APPLY: finished
0924 19:57:31.696596 (+ 215us) log.cc:844] Serialized 23 byte log entry
0924 19:57:31.696782 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:31.696853 (+ 71us) write_op.cc:454] Released schema lock
0924 19:57:31.696887 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:31.740735 (+ 43848us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:31.740766 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"spinlock_wait_cycles":62630656,"child_traces":[["op",{"apply.queue_time_us":293,"apply.run_cpu_time_us":1358,"apply.run_wall_time_us":3585,"num_ops":1,"prepare.queue_time_us":463828,"prepare.run_cpu_time_us":2051,"prepare.run_wall_time_us":62347,"raft.queue_time_us":182,"raft.run_cpu_time_us":716,"raft.run_wall_time_us":718,"replication_time_us":15671}]]}
W20260924 19:57:32.565706 17606 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 34) took 1228 ms (client timeout 1000 ms). Trace:
I20260924 19:57:32.570397 18348 ts_tablet_manager.cc:1916] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:32.571074 17944 tablet_service.cc:1511] Processing CreateTablet for tablet d970cc92a1134a98a613d198a319cb01 (DEFAULT_TABLE table=default.Table_a50429edeaf645668bc84244412bb359 [id=1853f611de514a55afd7a68d742503b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.579058 17944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d970cc92a1134a98a613d198a319cb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.524943 18539 leader_election.cc:290] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:32.541206 17753 leader_election.cc:304] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:32.552563 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet efd7339d0fee436299a7eac02a82addd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:32.555418 17797 tablet_service.cc:1558] Processing DeleteTablet for tablet 085b8088134d47478f0ce06a6a9f8572 with delete_type TABLET_DATA_DELETED (replaced by tablet fec9b6ecfb384b528cf71c4cfb2fd4cb) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:32.565404 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 916877c7540a4f6685064fc2807cb140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:32.580003 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077f70bcee7e412299da12ddc30a63ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.580176 18539 raft_consensus.cc:2804] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:32.580662 18539 raft_consensus.cc:493] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:32.567725 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet a384c9a0bb65429ebb97fe0807c78773 with delete_type TABLET_DATA_DELETED (replaced by tablet 077f70bcee7e412299da12ddc30a63ff) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:32.580998 18539 raft_consensus.cc:3060] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:32.534492 18116 raft_consensus.cc:2468] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:32.576617 17608 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 32466931871c4c5d8e9f95a11ffe26be
I20260924 19:57:32.585528 17608 catalog_manager.cc:6934] ReplaceTablet: tablet 32466931871c4c5d8e9f95a11ffe26be deleted and replaced by tablet affce4de8e0b412aa3b4bf19f9be7b6e
I20260924 19:57:32.586217 17972 raft_consensus.cc:2468] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:32.586941 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_bda24aa05e4c497385d8a40f8c9752b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:32.587209 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 32466931871c4c5d8e9f95a11ffe26be with delete_type TABLET_DATA_DELETED (replaced by tablet affce4de8e0b412aa3b4bf19f9be7b6e) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:32.587541 18220 raft_consensus.cc:359] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.588187 18220 raft_consensus.cc:385] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:32.588439 18220 raft_consensus.cc:740] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
W20260924 19:57:32.588423 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bda24aa05e4c497385d8a40f8c9752b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:32.589073 18220 consensus_queue.cc:260] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.589737 18539 raft_consensus.cc:515] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.592489 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2e4519aed7642678db3a518a89778de" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:32.592989 17967 raft_consensus.cc:3060] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:32.576830 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_954c770a550c49599e38ee3850e4aea1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:32.577915 17615 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b814c73599bb4d9fae83012912129997" } modify_external_catalogs: true
I20260924 19:57:32.595546 17615 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b814c73599bb4d9fae83012912129997" } modify_external_catalogs: true
I20260924 19:57:32.578831 17609 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_c9486a2257a84d9ab489210722683584" } new_table_name: "default.Table_5b3a0213ccbb4622a13a68e24d12af4e" modify_external_catalogs: true
I20260924 19:57:32.581868 17613 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_4d4391b7ade9428a8ed817a952ac87f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:32.602065 17613 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d4391b7ade9428a8ed817a952ac87f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:32.604867 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c884f12860164f12bbcd0d2f87067013" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:32.582609 17610 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_6115a46b8e7444e6899712d31f41390a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:32.606202 17969 raft_consensus.cc:2393] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eaf64bfb64aa42948c9bc1411ce4ce35 in current term 1: Already voted for candidate c4f6e3a7b06148918d230c77ab0c8202 in this term.
W20260924 19:57:32.607084 17610 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6115a46b8e7444e6899712d31f41390a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:32.607332 18220 ts_tablet_manager.cc:1434] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.066s user 0.004s sys 0.002s
I20260924 19:57:32.608255 17967 raft_consensus.cc:2468] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:32.611785 18220 tablet_bootstrap.cc:492] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:32.614292 17753 leader_election.cc:304] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:32.614467 18539 leader_election.cc:290] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:32.615015 18540 raft_consensus.cc:2804] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:32.615501 18540 raft_consensus.cc:697] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:32.616313 18540 consensus_queue.cc:237] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.620518 18231 ts_tablet_manager.cc:1916] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:32.621709 18090 tablet_service.cc:1511] Processing CreateTablet for tablet 2cf62a73c49245198d65011875148dc3 (DEFAULT_TABLE table=default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.631568 17753 leader_election.cc:304] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:32.632642 18536 raft_consensus.cc:2804] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 19:57:32.595094 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_954c770a550c49599e38ee3850e4aea1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:32.661418 18224 tablet_bootstrap.cc:654] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:32.665602 17949 tablet_service.cc:1511] Processing CreateTablet for tablet dbb82914df65416aaf47db8e507afb40 (DEFAULT_TABLE table=default.Table_79bc1edaa2024011a001b435d63329f4 [id=56af0067ee3548a29ee663929dfc9140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.436667 18117 rpcz_store.cc:276] 0924 19:57:31.131468 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.132798 (+ 1330us) service_pool.cc:224] Handling call
0924 19:57:32.291883 (+1159085us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:32.668352 17951 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a19b5812dc54589a5cbf48199951600 with delete_type TABLET_DATA_DELETED (replaced by tablet f38188eccc344246b72a8755ec5201ed) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:32.668958 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet 9b04059219084f608e8cb5fa4d0e0d58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:32.670614 17950 tablet_service.cc:1558] Processing DeleteTablet for tablet 151247d6b5254cd884c58db9311ac04a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:32.612165 17606 rpcz_store.cc:269] 0924 19:57:31.336960 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.508584 (+171624us) service_pool.cc:224] Handling call
0924 19:57:31.549581 (+ 40997us) spinlock_profiling.cc:243] Waited 36.7 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:32.565614 (+1016033us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:32.085071 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:32.085551 (+ 480us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:32.085588 (+ 37us) write_op.cc:435] Acquired schema lock
0924 19:57:32.085619 (+ 31us) tablet.cc:662] Decoding operations
0924 19:57:32.086043 (+ 424us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:32.086106 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:32.086128 (+ 22us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:32.086403 (+ 275us) tablet.cc:701] Row locks acquired
0924 19:57:32.086429 (+ 26us) write_op.cc:260] PREPARE: finished
0924 19:57:32.086642 (+ 213us) write_op.cc:270] Start()
0924 19:57:32.086775 (+ 133us) write_op.cc:276] Timestamp: P: 1790279852086461 usec, L: 0
0924 19:57:32.086814 (+ 39us) op_driver.cc:348] REPLICATION: starting
0924 19:57:32.087733 (+ 919us) log.cc:844] Serialized 530 byte log entry
0924 19:57:32.214681 (+126948us) op_driver.cc:464] REPLICATION: finished
0924 19:57:32.217102 (+ 2421us) write_op.cc:301] APPLY: starting
0924 19:57:32.217157 (+ 55us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:32.217250 (+ 93us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:32.217359 (+ 109us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:32.217545 (+ 186us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:32.217675 (+ 130us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:32.217715 (+ 40us) write_op.cc:312] APPLY: finished
0924 19:57:32.217900 (+ 185us) log.cc:844] Serialized 23 byte log entry
0924 19:57:32.218031 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:32.218089 (+ 58us) write_op.cc:454] Released schema lock
0924 19:57:32.218119 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:32.218337 (+ 218us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:32.218360 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"spinlock_wait_cycles":84472576,"child_traces":[["op",{"apply.queue_time_us":2229,"apply.run_cpu_time_us":1211,"apply.run_wall_time_us":1210,"num_ops":1,"prepare.queue_time_us":9902,"prepare.run_cpu_time_us":2991,"prepare.run_wall_time_us":3014,"replication_time_us":127783}]]}
I20260924 19:57:32.669942 17942 tablet_service.cc:1511] Processing CreateTablet for tablet 077f70bcee7e412299da12ddc30a63ff (DEFAULT_TABLE table=default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:57:32.681041 18050 leader_election.cc:343] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:32.679106 18107 raft_consensus.cc:2468] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:32.697391 17943 tablet_service.cc:1511] Processing CreateTablet for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb (DEFAULT_TABLE table=default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.705353 17606 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_a50429edeaf645668bc84244412bb359" } modify_external_catalogs: true
I20260924 19:57:32.706079 17606 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_a50429edeaf645668bc84244412bb359" } modify_external_catalogs: true
I20260924 19:57:32.707785 17946 tablet_service.cc:1558] Processing DeleteTablet for tablet a384c9a0bb65429ebb97fe0807c78773 with delete_type TABLET_DATA_DELETED (replaced by tablet 077f70bcee7e412299da12ddc30a63ff) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:32.711818 17948 tablet_service.cc:1511] Processing CreateTablet for tablet 0c08f10d2a364cb8a0a7ea40654c3fe4 (DEFAULT_TABLE table=default.Table_2bb6cac7492d4c5ca461d126788c9722 [id=f1f88aa92d884988b4bdf211754de8de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.721834 18087 tablet_service.cc:1558] Processing DeleteTablet for tablet 41bdb846ca19425b882434df0167479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:31 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:32.722601 18223 ts_tablet_manager.cc:1434] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.525s user 0.005s sys 0.010s
I20260924 19:57:32.727742 18231 ts_tablet_manager.cc:1929] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:32.724825 18089 tablet_service.cc:1511] Processing CreateTablet for tablet a765abd702644f079bddffb3fc4238ef (DEFAULT_TABLE table=default.Table_3ee32d095d774dbe88f86bff9878613b [id=8d4e859f129f43b2a9a8e12861b6b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.717767 18088 tablet_service.cc:1511] Processing CreateTablet for tablet 872bc32b9e384c828a0699079b27530d (DEFAULT_TABLE table=default.Table_3bf03540f35b468dacc7a86dd2d3d2a7 [id=eaf422968e464f1986f89b6944a6adbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.729176 18231 log.cc:1199] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/9c2440e1cade4327a63d6234e3adcd64
W20260924 19:57:32.722965 18223 ts_tablet_manager.cc:1467] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35: Tablet startup took 525ms
I20260924 19:57:32.734898 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c2440e1cade4327a63d6234e3adcd64 with delete_type TABLET_DATA_DELETED (replaced by tablet 07f86f24d9d14d05898e1c0f69428850) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:32.736312 17602 consensus_peers.cc:597] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e -> Peer 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309): Couldn't send request to peer 3e059bbe4dac4178bccb3be48cb1118d. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260924 19:57:32.745584 17906 leader_election.cc:304] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:32.747270 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet f38188eccc344246b72a8755ec5201ed (table default.Table_37482fb26ba144619bb66861dceaab3b [id=06b8b073d6ae475c8125613b89463d3c]) successfully deleted
W20260924 19:57:32.748047 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 41bdb846ca19425b882434df0167479e because the tablet was not found. No further retry: Not found: Tablet not found: 41bdb846ca19425b882434df0167479e
I20260924 19:57:32.740222 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb82914df65416aaf47db8e507afb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.751739 18231 ts_tablet_manager.cc:1950] T 9c2440e1cade4327a63d6234e3adcd64 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:32.755643 17942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077f70bcee7e412299da12ddc30a63ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.751341 18084 tablet_service.cc:1511] Processing CreateTablet for tablet 0c08f10d2a364cb8a0a7ea40654c3fe4 (DEFAULT_TABLE table=default.Table_2bb6cac7492d4c5ca461d126788c9722 [id=f1f88aa92d884988b4bdf211754de8de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.757879 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet a8cc802bf87f4d739175f93203478af8 with delete_type TABLET_DATA_DELETED (replaced by tablet a765abd702644f079bddffb3fc4238ef) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:32.759362 18504 raft_consensus.cc:2804] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:32.759930 18504 raft_consensus.cc:697] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:32.760527 18092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd6e656a653a43d1843a91fee813237d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.760869 18504 consensus_queue.cc:237] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.721869 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "349c6e9413bb41268d0c41d549203e2a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:32.739189 18085 tablet_service.cc:1511] Processing CreateTablet for tablet d970cc92a1134a98a613d198a319cb01 (DEFAULT_TABLE table=default.Table_a50429edeaf645668bc84244412bb359 [id=1853f611de514a55afd7a68d742503b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.758524 17946 tablet_service.cc:1558] Processing DeleteTablet for tablet 32466931871c4c5d8e9f95a11ffe26be with delete_type TABLET_DATA_DELETED (replaced by tablet affce4de8e0b412aa3b4bf19f9be7b6e) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:32.768961 18231 tablet_replica.cc:333] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:32.770133 17943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.772578 17948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c08f10d2a364cb8a0a7ea40654c3fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.780838 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd434e38c9364f55bd18383149015028" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:32.781250 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "151247d6b5254cd884c58db9311ac04a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:32.781800 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "151247d6b5254cd884c58db9311ac04a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:32.774636 18087 tablet_service.cc:1511] Processing CreateTablet for tablet c0feecd2fdb4409f86f6a5fbfbab3537 (DEFAULT_TABLE table=default.Table_5e50f3dbbb674d658a5afac0ebcedaa6 [id=545488f05d324b748e42dae4a665f0d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.784031 18109 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2e4519aed7642678db3a518a89778de" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:32.784528 18109 raft_consensus.cc:3060] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:32.784729 17946 tablet_service.cc:1511] Processing CreateTablet for tablet 3b328d37d55e4116a29322caaa21d431 (DEFAULT_TABLE table=default.Table_4e897947eb9e4466a9f050091b6a7b60 [id=b7032f43d384487d9b041986780669c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.786216 17946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b328d37d55e4116a29322caaa21d431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.788873 17950 tablet_service.cc:1511] Processing CreateTablet for tablet edc6c16578e5463a9d44c25fd6c538a9 (DEFAULT_TABLE table=default.Table_01931a7dc8a94582ae8c31c6078b66c5 [id=fa469a44fb3741d39036337129cc36a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.790126 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc6c16578e5463a9d44c25fd6c538a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.793388 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2e4519aed7642678db3a518a89778de" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:32.799376 17613 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_79bc1edaa2024011a001b435d63329f4" } new_table_name: "default.Table_152e3104b5364d4bba56f42d9491d56a" modify_external_catalogs: true
I20260924 19:57:32.802054 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c884f12860164f12bbcd0d2f87067013" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:32.812767 18116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a524bad1c4040e58ad84fcf5a4fc47d" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:32.814431 18110 raft_consensus.cc:3060] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:32.732303 18223 ts_tablet_manager.cc:1469] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35: Trace:
0924 19:57:30.223402 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:30.223503 (+ 101us) service_pool.cc:224] Handling call
0924 19:57:30.224859 (+ 1356us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:30.224940 (+ 81us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:30.234420 (+ 9480us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:30.252841 (+ 18421us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:30.262787 (+ 9946us) inbound_call.cc:177] Queueing success response
0924 19:57:31.560655 (+1297868us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0924 19:57:32.197701 (+637046us) ts_tablet_manager.cc:1435] Starting tablet replica
0924 19:57:32.281070 (+ 83369us) tablet_replica.cc:261] Starting instrumentation
0924 19:57:32.281546 (+ 476us) tablet_replica.cc:280] Starting consensus
Metrics: {"mutex_wait_us":3013,"raft.queue_time_us":107053,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":22341,"tablet-open.queue_time_us":1297949}
I20260924 19:57:32.769830 18536 raft_consensus.cc:697] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:32.850557 18224 tablet_bootstrap.cc:492] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:32.851029 18224 ts_tablet_manager.cc:1403] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.348s user 0.018s sys 0.004s
I20260924 19:57:32.853847 18224 raft_consensus.cc:359] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.854550 18224 raft_consensus.cc:385] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:32.854820 18224 raft_consensus.cc:740] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:32.856024 18536 consensus_queue.cc:237] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.866997 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 9c2440e1cade4327a63d6234e3adcd64 (table default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]) successfully deleted
W20260924 19:57:32.867769 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 8a19b5812dc54589a5cbf48199951600 because the tablet was not found. No further retry: Not found: Tablet not found: 8a19b5812dc54589a5cbf48199951600
W20260924 19:57:32.868319 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 9b04059219084f608e8cb5fa4d0e0d58 because the tablet was not found. No further retry: Not found: Tablet not found: 9b04059219084f608e8cb5fa4d0e0d58
I20260924 19:57:32.868690 17796 tablet_service.cc:1511] Processing CreateTablet for tablet 3b328d37d55e4116a29322caaa21d431 (DEFAULT_TABLE table=default.Table_4e897947eb9e4466a9f050091b6a7b60 [id=b7032f43d384487d9b041986780669c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.869751 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b328d37d55e4116a29322caaa21d431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.872218 17795 tablet_service.cc:1511] Processing CreateTablet for tablet edc6c16578e5463a9d44c25fd6c538a9 (DEFAULT_TABLE table=default.Table_01931a7dc8a94582ae8c31c6078b66c5 [id=fa469a44fb3741d39036337129cc36a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.873279 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc6c16578e5463a9d44c25fd6c538a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.875934 18578 raft_consensus.cc:493] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:32.877465 18578 raft_consensus.cc:515] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.885278 18426 consensus_queue.cc:1048] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:32.886098 18578 leader_election.cc:290] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:32.893280 17947 tablet_service.cc:1511] Processing CreateTablet for tablet dd2f4eb8bf254c4e85b6a9072271766b (DEFAULT_TABLE table=default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.894766 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2f4eb8bf254c4e85b6a9072271766b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.895936 17801 tablet_service.cc:1511] Processing CreateTablet for tablet dd2f4eb8bf254c4e85b6a9072271766b (DEFAULT_TABLE table=default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.897071 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2f4eb8bf254c4e85b6a9072271766b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.913548 18578 consensus_queue.cc:1048] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:32.918180 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0cf2df461598411f8ad9f3a51eafc6f7" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:32.919198 17610 master_service.cc:960] ReplaceTablet: received request to replace tablet a765abd702644f079bddffb3fc4238ef from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:32.893309 18348 ts_tablet_manager.cc:1929] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:32.922206 18348 log.cc:1199] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/9c2440e1cade4327a63d6234e3adcd64
I20260924 19:57:32.923213 18348 ts_tablet_manager.cc:1950] T 9c2440e1cade4327a63d6234e3adcd64 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:32.923054 17608 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 349c6e9413bb41268d0c41d549203e2a (table default.Table_41c56bb86db34f16aba6e94147cde1d2 [id=b1d9f8b4039b4d19bfc493355e0d07cc]). Expected version 1 got 0
I20260924 19:57:32.924070 17608 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_56ddb56503844349ba42c68ebb67a694 [id=522c4df3169f4d2ba1cbe15df69d40b3]). Expected version 1 got 0
I20260924 19:57:32.925454 18348 tablet_replica.cc:333] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:32.931648 18086 tablet_service.cc:1511] Processing CreateTablet for tablet dbb82914df65416aaf47db8e507afb40 (DEFAULT_TABLE table=default.Table_79bc1edaa2024011a001b435d63329f4 [id=56af0067ee3548a29ee663929dfc9140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.933179 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 9c2440e1cade4327a63d6234e3adcd64 (table default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]) successfully deleted
W20260924 19:57:32.933992 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 8a19b5812dc54589a5cbf48199951600 because the tablet was not found. No further retry: Not found: Tablet not found: 8a19b5812dc54589a5cbf48199951600
W20260924 19:57:32.934620 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 9b04059219084f608e8cb5fa4d0e0d58 because the tablet was not found. No further retry: Not found: Tablet not found: 9b04059219084f608e8cb5fa4d0e0d58
W20260924 19:57:32.944869 17753 leader_election.cc:343] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:32.948537 17612 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:32.953236 18540 consensus_queue.cc:1048] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:32.953492 18224 consensus_queue.cc:260] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:32.958348 18220 tablet_bootstrap.cc:654] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:32.951573 17606 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet d970cc92a1134a98a613d198a319cb01
I20260924 19:57:32.966493 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 89) took 1746 ms. Trace:
I20260924 19:57:32.982225 17612 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 5f994e0401df4fc19bc5abb10f08bc6e (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]). Expected version 1 got 0
I20260924 19:57:32.983111 17612 catalog_manager.cc:5838] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:32.985277 17612 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 97a1b8ccd35a43cea7a71d705fad1494 (table default.Table_850e5158d78d4850b87445dac75ef7a9 [id=f768ce157a7b407d9f1e55a5b9d0720a]). Expected version 1 got 0
I20260924 19:57:32.986052 17612 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 1 got 0
I20260924 19:57:32.990391 18083 tablet_service.cc:1511] Processing CreateTablet for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb (DEFAULT_TABLE table=default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:32.993822 18093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1684aaf9d984649bff8c4b0b12044b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.000016 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41bdb846ca19425b882434df0167479e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.004194 18097 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 92) took 1056 ms. Trace:
I20260924 19:57:33.004525 18097 rpcz_store.cc:276] 0924 19:57:31.945180 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.951921 (+ 6741us) service_pool.cc:224] Handling call
0924 19:57:32.132372 (+180451us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:32.153857 (+ 21485us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:32.258763 (+104906us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:32.657131 (+398368us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:33.000910 (+343779us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1128064}
I20260924 19:57:33.005568 18091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c861e93ad9f40ba9ad010021e43bc2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:32.968711 18051 leader_election.cc:304] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:32.969321 18115 raft_consensus.cc:2468] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:33.010741 18083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.012053 18348 raft_consensus.cc:2243] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:33.012612 18348 raft_consensus.cc:2272] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:33.013187 17944 tablet_service.cc:1558] Processing DeleteTablet for tablet d970cc92a1134a98a613d198a319cb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:33.014616 18348 ts_tablet_manager.cc:1916] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:33.015638 17612 catalog_manager.cc:5838] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:33.017328 17612 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:33.017970 17612 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_56ddb56503844349ba42c68ebb67a694 [id=522c4df3169f4d2ba1cbe15df69d40b3]). Expected version 1 got 0
I20260924 19:57:33.018787 17612 catalog_manager.cc:5838] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
W20260924 19:57:33.021631 18050 leader_election.cc:343] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Not found: Tablet not found: f38188eccc344246b72a8755ec5201ed
I20260924 19:57:33.022244 18231 raft_consensus.cc:2243] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:33.022967 18088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872bc32b9e384c828a0699079b27530d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.023873 18231 pending_rounds.cc:70] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: Trying to abort 1 pending ops.
I20260924 19:57:33.024168 18231 pending_rounds.cc:77] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986277521940480 op_type: NO_OP noop_request { }
I20260924 19:57:33.024614 18231 raft_consensus.cc:2889] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:33.025017 18231 raft_consensus.cc:2272] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:33.025168 17612 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 349c6e9413bb41268d0c41d549203e2a (table default.Table_41c56bb86db34f16aba6e94147cde1d2 [id=b1d9f8b4039b4d19bfc493355e0d07cc]). Expected version 1 got 0
I20260924 19:57:33.033367 17615 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet f482b53a4a2d44f6b3431c588cd605ce
I20260924 19:57:33.039618 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet d970cc92a1134a98a613d198a319cb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:32.996330 18109 raft_consensus.cc:2468] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:32.957731 18224 ts_tablet_manager.cc:1434] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.106s user 0.005s sys 0.001s
I20260924 19:57:33.045094 18561 raft_consensus.cc:2749] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:33.045805 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 151247d6b5254cd884c58db9311ac04a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:33.046243 18224 tablet_bootstrap.cc:492] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:33.049356 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 80730.3us
I20260924 19:57:33.056260 18051 leader_election.cc:304] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:33.056807 18320 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:57:33.057240 18320 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
I20260924 19:57:33.057453 18561 raft_consensus.cc:2749] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:33.057678 18320 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:57:33.058171 18320 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:57:33.058985 17753 leader_election.cc:343] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:33.059684 17753 leader_election.cc:304] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:33.061630 18540 raft_consensus.cc:2749] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:33.058429 18085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d970cc92a1134a98a613d198a319cb01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.067448 18223 tablet_bootstrap.cc:492] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:33.070399 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet efd7339d0fee436299a7eac02a82addd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:33.073700 18095 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 94) took 1127 ms. Trace:
I20260924 19:57:33.074014 18095 rpcz_store.cc:276] 0924 19:57:31.945901 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.950517 (+ 4616us) service_pool.cc:224] Handling call
0924 19:57:32.151559 (+201042us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c606a02 00007fb84554fd19 00007fb84478ba70 00007fb844870e61 00007fb84484e0f3 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82 00007fb842d86ab1
0924 19:57:32.254769 (+103210us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606e7c 00007fb83c606a3b 00007fb84554fd19 00007fb84478ba70 00007fb844870e61 00007fb84484e0f3 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82 00007fb842d86ab1
0924 19:57:32.353090 (+ 98321us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:32.353171 (+ 81us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:32.967295 (+614124us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:33.009761 (+ 42466us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:33.073635 (+ 63874us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60802560}
I20260924 19:57:32.992655 18492 rpcz_store.cc:276] 0924 19:57:31.220363 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.224871 (+ 4508us) service_pool.cc:224] Handling call
0924 19:57:31.225722 (+ 851us) ts_tablet_manager.cc:1151] Deleting tablet a384c9a0bb65429ebb97fe0807c78773
0924 19:57:31.225818 (+ 96us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:31.559917 (+334099us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:31.689337 (+129420us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:32.966416 (+1277079us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24399,"tablet-delete.queue_time_us":237}
W20260924 19:57:33.082659 17753 leader_election.cc:343] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:33.087567 17609 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c884f12860164f12bbcd0d2f87067013 consensus state has no leader
I20260924 19:57:33.088099 17609 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 152) took 1092 ms. Trace:
I20260924 19:57:33.071657 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0cf2df461598411f8ad9f3a51eafc6f7" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:33.099535 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet f482b53a4a2d44f6b3431c588cd605ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:33.088366 17609 rpcz_store.cc:276] 0924 19:57:31.995539 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.501273 (+505734us) service_pool.cc:224] Handling call
0924 19:57:32.578750 (+ 77477us) spinlock_profiling.cc:243] Waited 76.8 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa5522077e6 00007fa54fdbc634 00007fa54fdbc5c2 00007fa54fdbc4f1 00007fa54fdbc46c 00007fa54fdbb722 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:32.597203 (+ 18453us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:32.598029 (+ 826us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:32.599066 (+ 1037us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:32.599360 (+ 294us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:32.717316 (+117956us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:33.088030 (+370714us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:32.609193 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:32.609550 (+ 357us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:32.609572 (+ 22us) write_op.cc:435] Acquired schema lock
0924 19:57:32.609594 (+ 22us) tablet.cc:662] Decoding operations
0924 19:57:32.611265 (+ 1671us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:32.611299 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:32.611318 (+ 19us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:32.611468 (+ 150us) tablet.cc:701] Row locks acquired
0924 19:57:32.611480 (+ 12us) write_op.cc:260] PREPARE: finished
0924 19:57:32.611582 (+ 102us) write_op.cc:270] Start()
0924 19:57:32.611656 (+ 74us) write_op.cc:276] Timestamp: P: 1790279852611530 usec, L: 0
0924 19:57:32.611670 (+ 14us) op_driver.cc:348] REPLICATION: starting
0924 19:57:32.612122 (+ 452us) log.cc:844] Serialized 405 byte log entry
0924 19:57:32.704442 (+ 92320us) op_driver.cc:464] REPLICATION: finished
0924 19:57:32.707172 (+ 2730us) write_op.cc:301] APPLY: starting
0924 19:57:32.707244 (+ 72us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:32.707328 (+ 84us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:32.707416 (+ 88us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:32.707572 (+ 156us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:32.707694 (+ 122us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:32.707735 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:57:32.707947 (+ 212us) log.cc:844] Serialized 23 byte log entry
0924 19:57:32.708068 (+ 121us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:32.708122 (+ 54us) write_op.cc:454] Released schema lock
0924 19:57:32.708156 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:32.712233 (+ 4077us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:32.713812 (+ 1579us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":176570752,"child_traces":[["op",{"apply.queue_time_us":2575,"apply.run_cpu_time_us":1187,"apply.run_wall_time_us":1183,"num_ops":1,"prepare.queue_time_us":8469,"prepare.run_cpu_time_us":1870,"prepare.run_wall_time_us":103031,"raft.queue_time_us":80,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":259,"replication_time_us":92699}]]}
I20260924 19:57:33.112766 17606 master_service.cc:960] ReplaceTablet: received request to replace tablet a765abd702644f079bddffb3fc4238ef from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:33.115491 18220 tablet_bootstrap.cc:492] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:33.115914 18220 ts_tablet_manager.cc:1403] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.504s user 0.008s sys 0.010s
W20260924 19:57:33.117532 17607 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 32) took 1471 ms (client timeout 1000 ms). Trace:
W20260924 19:57:33.136312 17753 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.17.2.189:43689, user_credentials={real_user=slave}} blocked reactor thread for 52064.4us
W20260924 19:57:33.118026 17607 rpcz_store.cc:269] 0924 19:57:31.645782 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.711788 (+ 66006us) service_pool.cc:224] Handling call
0924 19:57:31.833958 (+122170us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:33.117470 (+1283512us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:32.572819 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:32.606155 (+ 33336us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:32.606194 (+ 39us) write_op.cc:435] Acquired schema lock
0924 19:57:32.606208 (+ 14us) tablet.cc:662] Decoding operations
0924 19:57:32.606619 (+ 411us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:32.606660 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:32.606683 (+ 23us) tablet.cc:685] Acquiring locks for 3 operations
0924 19:57:32.606930 (+ 247us) tablet.cc:701] Row locks acquired
0924 19:57:32.606944 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:32.607055 (+ 111us) write_op.cc:270] Start()
0924 19:57:32.607147 (+ 92us) write_op.cc:276] Timestamp: P: 1790279852607002 usec, L: 0
0924 19:57:32.607186 (+ 39us) op_driver.cc:348] REPLICATION: starting
0924 19:57:32.607749 (+ 563us) log.cc:844] Serialized 1284 byte log entry
0924 19:57:32.704182 (+ 96433us) op_driver.cc:464] REPLICATION: finished
0924 19:57:32.705278 (+ 1096us) write_op.cc:301] APPLY: starting
0924 19:57:32.705333 (+ 55us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:32.705470 (+ 137us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:32.705569 (+ 99us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:32.705922 (+ 353us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:32.706067 (+ 145us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0924 19:57:32.706115 (+ 48us) write_op.cc:312] APPLY: finished
0924 19:57:32.706323 (+ 208us) log.cc:844] Serialized 35 byte log entry
0924 19:57:32.706496 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:32.706571 (+ 75us) write_op.cc:454] Released schema lock
0924 19:57:32.706607 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:32.706848 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:32.706870 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":14,"spinlock_wait_cycles":274829056,"child_traces":[["op",{"apply.queue_time_us":917,"apply.run_cpu_time_us":1536,"apply.run_wall_time_us":1535,"num_ops":3,"prepare.queue_time_us":115643,"prepare.run_cpu_time_us":2239,"prepare.run_wall_time_us":36249,"raft.queue_time_us":4137,"raft.run_cpu_time_us":971,"raft.run_wall_time_us":46296,"replication_time_us":96888}]]}
I20260924 19:57:33.099850 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf62a73c49245198d65011875148dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.145107 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet 916877c7540a4f6685064fc2807cb140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:33.148900 18231 ts_tablet_manager.cc:1916] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:33.162644 18348 ts_tablet_manager.cc:1929] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:33.163137 18348 log.cc:1199] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/151247d6b5254cd884c58db9311ac04a
I20260924 19:57:33.163856 18348 ts_tablet_manager.cc:1950] T 151247d6b5254cd884c58db9311ac04a P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:33.168862 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_a82c256e36e448c296c86a30176f4135"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:33.169839 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_fe4242bbbc1f4747bd1f2b4d25c8bacd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:33.170387 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a82c256e36e448c296c86a30176f4135 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:33.173660 18220 raft_consensus.cc:359] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.174008 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_1b2f1b739e234953861c274a098c2e1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:33.174391 18220 raft_consensus.cc:385] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:33.174654 18220 raft_consensus.cc:740] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
W20260924 19:57:33.175567 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b2f1b739e234953861c274a098c2e1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:33.179801 17613 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbb82914df65416aaf47db8e507afb40 consensus state has no leader
I20260924 19:57:33.105780 18094 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 93) took 1130 ms. Trace:
I20260924 19:57:33.186892 18220 consensus_queue.cc:260] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.213238 18573 raft_consensus.cc:493] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:33.213748 18573 raft_consensus.cc:515] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.215185 18220 ts_tablet_manager.cc:1434] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.075s user 0.007s sys 0.000s
I20260924 19:57:33.215883 18573 leader_election.cc:290] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:33.226869 18220 tablet_bootstrap.cc:492] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
W20260924 19:57:33.231320 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 132738us
I20260924 19:57:33.236545 17943 tablet_service.cc:1558] Processing DeleteTablet for tablet f482b53a4a2d44f6b3431c588cd605ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:33.239842 18348 tablet_replica.cc:333] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:33.241303 17948 tablet_service.cc:1558] Processing DeleteTablet for tablet 916877c7540a4f6685064fc2807cb140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:33.241669 17943 tablet_service.cc:1558] Processing DeleteTablet for tablet 085b8088134d47478f0ce06a6a9f8572 with delete_type TABLET_DATA_DELETED (replaced by tablet fec9b6ecfb384b528cf71c4cfb2fd4cb) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:33.241039 17946 tablet_service.cc:1558] Processing DeleteTablet for tablet efd7339d0fee436299a7eac02a82addd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:33.247210 17946 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c2440e1cade4327a63d6234e3adcd64 with delete_type TABLET_DATA_DELETED (replaced by tablet 07f86f24d9d14d05898e1c0f69428850) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:33.248178 18348 raft_consensus.cc:2243] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:33.248919 18348 raft_consensus.cc:2272] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:33.148504 17753 leader_election.cc:304] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:33.251873 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 103453us
I20260924 19:57:33.251956 18573 raft_consensus.cc:2804] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:33.252405 18573 raft_consensus.cc:493] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:33.252717 18573 raft_consensus.cc:3060] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:33.253211 17753 leader_election.cc:343] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:33.253695 17753 leader_election.cc:304] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:33.255208 18224 tablet_bootstrap.cc:654] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:33.260104 18573 raft_consensus.cc:515] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.262279 18573 leader_election.cc:290] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:33.267289 18348 ts_tablet_manager.cc:1916] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:33.269661 18578 raft_consensus.cc:2749] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:33.271529 18536 consensus_queue.cc:1048] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:33.268528 18220 tablet_bootstrap.cc:654] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:33.273442 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "910d61b8c3574302b5a328187226571e" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:33.274931 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 077f70bcee7e412299da12ddc30a63ff (DEFAULT_TABLE table=default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:33.280735 18224 tablet_bootstrap.cc:492] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:33.281302 18224 ts_tablet_manager.cc:1403] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.235s user 0.011s sys 0.000s
I20260924 19:57:33.281478 18095 tablet_service.cc:1511] Processing CreateTablet for tablet dd2f4eb8bf254c4e85b6a9072271766b (DEFAULT_TABLE table=default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:57:33.266301 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 210173us
I20260924 19:57:33.283789 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd434e38c9364f55bd18383149015028" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:33.283947 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "910d61b8c3574302b5a328187226571e" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:33.283891 18224 raft_consensus.cc:359] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.284456 18224 raft_consensus.cc:385] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:33.284708 18224 raft_consensus.cc:740] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:33.277051 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbb82914df65416aaf47db8e507afb40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.285293 18224 consensus_queue.cc:260] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.302580 18224 ts_tablet_manager.cc:1434] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.021s user 0.001s sys 0.005s
I20260924 19:57:33.305727 18224 tablet_bootstrap.cc:492] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:33.306942 17753 leader_election.cc:343] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:33.309062 18220 tablet_bootstrap.cc:492] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:33.309243 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 32466931871c4c5d8e9f95a11ffe26be with delete_type TABLET_DATA_DELETED (replaced by tablet affce4de8e0b412aa3b4bf19f9be7b6e) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:33.309620 17753 leader_election.cc:343] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:33.310180 18220 ts_tablet_manager.cc:1403] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.093s user 0.010s sys 0.000s
I20260924 19:57:33.312709 18220 raft_consensus.cc:359] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.186271 18094 rpcz_store.cc:276] 0924 19:57:31.945572 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.950517 (+ 4945us) service_pool.cc:224] Handling call
0924 19:57:32.152007 (+201490us) spinlock_profiling.cc:243] Waited 5.77 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c606a02 00007fb84554fd19 00007fb84478ba70 00007fb844870e61 00007fb84484e0f3 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82 00007fb842d86ab1
0924 19:57:32.291454 (+139447us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606e7c 00007fb83c606a3b 00007fb84554fd19 00007fb84478ba70 00007fb844870e61 00007fb84484e0f3 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82 00007fb842d86ab1
0924 19:57:32.423493 (+132039us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:32.438687 (+ 15194us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:32.674613 (+235926us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:32.881084 (+206471us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:33.075674 (+194590us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39436800}
I20260924 19:57:33.329648 18224 tablet_bootstrap.cc:654] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:33.328151 18097 tablet_service.cc:1511] Processing CreateTablet for tablet 3b328d37d55e4116a29322caaa21d431 (DEFAULT_TABLE table=default.Table_4e897947eb9e4466a9f050091b6a7b60 [id=b7032f43d384487d9b041986780669c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:33.313426 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c08f10d2a364cb8a0a7ea40654c3fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.300467 18492 tablet_replica.cc:333] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:33.311714 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd434e38c9364f55bd18383149015028" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
W20260924 19:57:33.336273 17753 leader_election.cc:343] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:33.340917 18080 tablet_service.cc:1558] Processing DeleteTablet for tablet f482b53a4a2d44f6b3431c588cd605ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:33.339128 18082 tablet_service.cc:1511] Processing CreateTablet for tablet edc6c16578e5463a9d44c25fd6c538a9 (DEFAULT_TABLE table=default.Table_01931a7dc8a94582ae8c31c6078b66c5 [id=fa469a44fb3741d39036337129cc36a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:33.339993 18600 tablet.cc:1751] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:33.343207 17753 leader_election.cc:304] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:33.343942 18540 raft_consensus.cc:2749] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:33.343868 18079 tablet_service.cc:1558] Processing DeleteTablet for tablet a2e4519aed7642678db3a518a89778de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:33.332058 18601 tablet.cc:1751] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:33.344507 18220 raft_consensus.cc:385] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:33.344784 18220 raft_consensus.cc:740] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:33.345489 18573 consensus_queue.cc:1048] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:33.339550 18081 tablet_service.cc:1558] Processing DeleteTablet for tablet d970cc92a1134a98a613d198a319cb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:33.347427 18573 consensus_queue.cc:1048] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:33.194698 17613 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_41c56bb86db34f16aba6e94147cde1d2" } modify_external_catalogs: true
I20260924 19:57:33.354902 17613 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_41c56bb86db34f16aba6e94147cde1d2" } modify_external_catalogs: true
I20260924 19:57:33.354895 18220 consensus_queue.cc:260] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:33.242586 17601 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_152e3104b5364d4bba56f42d9491d56a [id=56af0067ee3548a29ee663929dfc9140], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dbb82914df65416aaf47db8e507afb40 consensus state has no leader
I20260924 19:57:33.247059 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 151247d6b5254cd884c58db9311ac04a (table default.Table_c8e27940040c495780f92f92431541cb [id=47a10be86ad441a7961c0f34fed46040]) successfully deleted
W20260924 19:57:33.357288 17599 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 110240us
W20260924 19:57:33.358659 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet a384c9a0bb65429ebb97fe0807c78773 because the tablet was not found. No further retry: Not found: Tablet not found: a384c9a0bb65429ebb97fe0807c78773
W20260924 19:57:33.249935 17612 catalog_manager.cc:6038] Failed to send AlterTable RPC for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 8a524bad1c4040e58ad84fcf5a4fc47d consensus state has no leader
I20260924 19:57:33.364712 17796 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c2440e1cade4327a63d6234e3adcd64 with delete_type TABLET_DATA_DELETED (replaced by tablet 07f86f24d9d14d05898e1c0f69428850) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:33.366207 18224 tablet_bootstrap.cc:492] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:33.366618 18224 ts_tablet_manager.cc:1403] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.003s
I20260924 19:57:33.367837 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet a2e4519aed7642678db3a518a89778de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:33.362527 17612 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 35) took 943 ms (client timeout 1000 ms). Trace:
I20260924 19:57:33.326584 17606 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_152e3104b5364d4bba56f42d9491d56a" } new_table_name: "default.Table_e2d6f648c21f44ad9674d28e00de881e" modify_external_catalogs: true
I20260924 19:57:33.368996 18224 raft_consensus.cc:359] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.369629 18224 raft_consensus.cc:385] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:33.369886 18224 raft_consensus.cc:740] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:33.370532 18224 consensus_queue.cc:260] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.196786 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_4df831f35ae74b2f86e7e1b653b69867"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:33.372582 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4df831f35ae74b2f86e7e1b653b69867 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:33.227725 17610 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet a765abd702644f079bddffb3fc4238ef
I20260924 19:57:33.376351 17610 catalog_manager.cc:6934] ReplaceTablet: tablet a765abd702644f079bddffb3fc4238ef deleted and replaced by tablet f169c47d711d4566b2004cdfd0de725d
I20260924 19:57:33.300830 17614 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_412220a60c3e4a16a5c4abcbbc48357c" } modify_external_catalogs: true
I20260924 19:57:33.378489 17614 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_412220a60c3e4a16a5c4abcbbc48357c" } modify_external_catalogs: true
I20260924 19:57:33.345839 18118 raft_consensus.cc:3060] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:33.387662 18224 ts_tablet_manager.cc:1434] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260924 19:57:33.388608 18224 tablet_bootstrap.cc:492] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:33.388518 17610 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_3ee32d095d774dbe88f86bff9878613b" } new_table_name: "default.Table_71fe66684fd042fb9d84f22528532866" modify_external_catalogs: true
W20260924 19:57:33.181854 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe4242bbbc1f4747bd1f2b4d25c8bacd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:33.397717 17946 tablet_service.cc:1558] Processing DeleteTablet for tablet a765abd702644f079bddffb3fc4238ef with delete_type TABLET_DATA_DELETED (replaced by tablet f169c47d711d4566b2004cdfd0de725d) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:33.402187 18224 tablet_bootstrap.cc:654] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:33.403411 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet a765abd702644f079bddffb3fc4238ef with delete_type TABLET_DATA_DELETED (replaced by tablet f169c47d711d4566b2004cdfd0de725d) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:33.411715 17612 rpcz_store.cc:269] 0924 19:57:32.418961 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.495613 (+ 76652us) service_pool.cc:224] Handling call
0924 19:57:32.573930 (+ 78317us) spinlock_profiling.cc:243] Waited 75.2 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:33.362468 (+788538us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:33.110218 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:33.110630 (+ 412us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:33.110656 (+ 26us) write_op.cc:435] Acquired schema lock
0924 19:57:33.110669 (+ 13us) tablet.cc:662] Decoding operations
0924 19:57:33.111130 (+ 461us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:33.111185 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:33.111208 (+ 23us) tablet.cc:685] Acquiring locks for 3 operations
0924 19:57:33.111464 (+ 256us) tablet.cc:701] Row locks acquired
0924 19:57:33.111478 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:33.111592 (+ 114us) write_op.cc:270] Start()
0924 19:57:33.111673 (+ 81us) write_op.cc:276] Timestamp: P: 1790279853111483 usec, L: 0
0924 19:57:33.111688 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:57:33.172316 (+ 60628us) log.cc:844] Serialized 1284 byte log entry
0924 19:57:33.173445 (+ 1129us) spinlock_profiling.cc:243] Waited 60.1 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f7aa1 00007fa55038a5e8 00007fa55081a741 00007fa550819f18 00007fa55081ebb1 00007fa55081eb69 00007fa55081eaf9 00007fa55081eac1 00007fa55081dced 00000000005bd4e4 00000000005bd319
0924 19:57:33.192000 (+ 18555us) op_driver.cc:464] REPLICATION: finished
0924 19:57:33.205092 (+ 13092us) write_op.cc:301] APPLY: starting
0924 19:57:33.205148 (+ 56us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:33.205280 (+ 132us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:33.205300 (+ 20us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:33.205629 (+ 329us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:33.205779 (+ 150us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0924 19:57:33.205824 (+ 45us) write_op.cc:312] APPLY: finished
0924 19:57:33.206020 (+ 196us) log.cc:844] Serialized 35 byte log entry
0924 19:57:33.206161 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:33.206232 (+ 71us) write_op.cc:454] Released schema lock
0924 19:57:33.206271 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:33.207374 (+ 1103us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:33.207399 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":14220,"reported_tablets":20,"spinlock_wait_cycles":172894080,"child_traces":[["op",{"apply.queue_time_us":12929,"apply.run_cpu_time_us":1397,"apply.run_wall_time_us":2253,"num_ops":3,"prepare.queue_time_us":106,"prepare.run_cpu_time_us":3899,"prepare.run_wall_time_us":63457,"raft.queue_time_us":25704,"raft.run_cpu_time_us":1028,"raft.run_wall_time_us":17193,"replication_time_us":80230,"spinlock_wait_cycles":138188672}]]}
W20260924 19:57:33.427788 18110 consensus_meta.cc:287] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent flushing consensus metadata: real 0.613s user 0.004s sys 0.004s
I20260924 19:57:33.428120 18110 raft_consensus.cc:2468] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:33.428634 18110 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.17.2.129:42273 (request call id 34) took 1308 ms. Trace:
I20260924 19:57:33.428612 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_aeac9df2b26b490a83f10d672335d75e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:33.434815 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeac9df2b26b490a83f10d672335d75e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:33.428918 18110 rpcz_store.cc:276] 0924 19:57:32.138933 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.178496 (+ 39563us) service_pool.cc:224] Handling call
0924 19:57:32.340557 (+162061us) spinlock_profiling.cc:243] Waited 5.92 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c606a02 00007fb84554fd19 00007fb84478ba70 00007fb8448546db 00007fb844854761 00007fb8448022e0 00007fb844801bc4 00007fb842e81834 00007fb842e817c2
0924 19:57:32.623144 (+282587us) spinlock_profiling.cc:243] Waited 1.63 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c606ece 00007fb845b05179 00007fb84478bd70 00007fb8448546f4 00007fb844854761 00007fb8448022e0 00007fb844801bc4 00007fb842e81834 00007fb842e817c2
0924 19:57:33.428574 (+805430us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17366272}
I20260924 19:57:33.456746 18223 tablet_bootstrap.cc:654] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:33.475600 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0feecd2fdb4409f86f6a5fbfbab3537. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.479525 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b328d37d55e4116a29322caaa21d431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.479789 18220 ts_tablet_manager.cc:1434] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.169s user 0.006s sys 0.000s
I20260924 19:57:33.306289 18051 leader_election.cc:304] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:33.496290 18220 tablet_bootstrap.cc:492] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:33.499411 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a765abd702644f079bddffb3fc4238ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.517628 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edc6c16578e5463a9d44c25fd6c538a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.518867 18348 ts_tablet_manager.cc:1929] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:33.521721 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2f4eb8bf254c4e85b6a9072271766b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.522224 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 077f70bcee7e412299da12ddc30a63ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.525961 18348 log.cc:1199] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/32466931871c4c5d8e9f95a11ffe26be
I20260924 19:57:33.527473 18348 ts_tablet_manager.cc:1950] T 32466931871c4c5d8e9f95a11ffe26be P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:33.531869 18231 ts_tablet_manager.cc:1929] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:33.524534 18220 tablet_bootstrap.cc:654] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:33.546391 18220 tablet_bootstrap.cc:492] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:33.546864 18220 ts_tablet_manager.cc:1403] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.003s
I20260924 19:57:33.550866 18348 tablet_replica.cc:333] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:33.552541 18348 raft_consensus.cc:2243] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:33.557718 18223 tablet_bootstrap.cc:492] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:33.558126 18223 ts_tablet_manager.cc:1403] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.491s user 0.009s sys 0.004s
I20260924 19:57:33.559047 18348 raft_consensus.cc:2272] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:33.560563 18223 raft_consensus.cc:359] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.561169 18223 raft_consensus.cc:385] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:33.561425 18223 raft_consensus.cc:740] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:33.562048 18223 consensus_queue.cc:260] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.564885 18348 ts_tablet_manager.cc:1916] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:33.585590 18492 raft_consensus.cc:2243] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260924 19:57:33.569758 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 268562us
W20260924 19:57:33.586576 18051 leader_election.cc:343] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: f38188eccc344246b72a8755ec5201ed
I20260924 19:57:33.586946 18051 leader_election.cc:304] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:33.589855 18093 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 97) took 1389 ms. Trace:
I20260924 19:57:33.590310 18093 rpcz_store.cc:276] 0924 19:57:32.180658 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.235038 (+ 54380us) service_pool.cc:224] Handling call
0924 19:57:32.988652 (+753614us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:32.990703 (+ 2051us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:33.089652 (+ 98949us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:33.485822 (+396170us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:33.570233 (+ 84411us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11157}
I20260924 19:57:33.591636 18426 raft_consensus.cc:493] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:33.592181 18426 raft_consensus.cc:515] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.594304 18426 leader_election.cc:290] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:33.566334 18220 raft_consensus.cc:359] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.619469 18096 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 95) took 1504 ms. Trace:
I20260924 19:57:33.619859 18096 rpcz_store.cc:276] 0924 19:57:32.115171 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.130360 (+ 15189us) service_pool.cc:224] Handling call
0924 19:57:32.254470 (+124110us) spinlock_profiling.cc:243] Waited 6.5 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c606a02 00007fb84554fd19 00007fb84478ba70 00007fb844870e61 00007fb84484e0f3 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82 00007fb842d86ab1
0924 19:57:32.352702 (+ 98232us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:32.352813 (+ 111us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:32.861063 (+508250us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:33.174600 (+313537us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:33.564446 (+389846us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c607182 00007fb845578ec9 00007fb84478b918 00007fb844851fa3 00007fb84484dd3f 00007fb84484e4fd 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82
0924 19:57:33.619336 (+ 54890us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":411060352}
I20260924 19:57:33.620970 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 32466931871c4c5d8e9f95a11ffe26be (table default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06]) successfully deleted
W20260924 19:57:33.621693 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet d970cc92a1134a98a613d198a319cb01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d970cc92a1134a98a613d198a319cb01 already in progress: creating tablet
I20260924 19:57:33.626801 18091 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 99) took 1347 ms. Trace:
I20260924 19:57:33.627377 18091 rpcz_store.cc:276] 0924 19:57:32.279492 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.298730 (+ 19238us) service_pool.cc:224] Handling call
0924 19:57:32.990736 (+692006us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:32.995532 (+ 4796us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:33.414131 (+418599us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:33.574472 (+160341us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:33.626741 (+ 52269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:33.619078 18220 raft_consensus.cc:385] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:33.630627 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_771765c449ce492986addb028e1b11d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:33.630127 18220 raft_consensus.cc:740] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
W20260924 19:57:33.632344 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_771765c449ce492986addb028e1b11d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:33.640637 18426 raft_consensus.cc:493] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:33.641179 18426 raft_consensus.cc:515] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.653247 18426 leader_election.cc:290] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:33.635705 18220 consensus_queue.cc:260] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.669087 18220 ts_tablet_manager.cc:1434] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.122s user 0.006s sys 0.000s
I20260924 19:57:33.682363 17607 master_service.cc:960] ReplaceTablet: received request to replace tablet ef3f82de70bb4328a15a044fe0e500bf from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:33.685485 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet a384c9a0bb65429ebb97fe0807c78773 (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]) successfully deleted
I20260924 19:57:33.570395 18578 consensus_queue.cc:1048] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260924 19:57:33.691615 18231 log.cc:1199] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/151247d6b5254cd884c58db9311ac04a
I20260924 19:57:33.593333 18224 tablet_bootstrap.cc:492] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:33.692423 18231 ts_tablet_manager.cc:1950] T 151247d6b5254cd884c58db9311ac04a P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:33.692502 18224 ts_tablet_manager.cc:1403] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.304s user 0.046s sys 0.046s
I20260924 19:57:33.693193 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 103) took 1193 ms. Trace:
I20260924 19:57:33.693502 18231 rpcz_store.cc:276] 0924 19:57:32.499541 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.512581 (+ 13040us) service_pool.cc:224] Handling call
0924 19:57:32.768776 (+256195us) ts_tablet_manager.cc:1151] Deleting tablet 151247d6b5254cd884c58db9311ac04a
0924 19:57:32.768823 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:33.023346 (+254523us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x7b5400111a54. stack: 00007f7c8a13c323 00007f7c8a13c220 00007f7c89ce0574 0000000000516964 00000000005168f9 00007f7c91239aa1 00007f7c912d8b22 00007f7c9171950c 00007f7c92382c3a 00007f7c923b3960 00007f7c92399e6d 00007f7c92399e19 00007f7c92399da9 00007f7c92399d71 00007f7c92398f9d 00000000005bd4e4
0924 19:57:33.162499 (+139153us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:33.531689 (+369190us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:33.693134 (+161445us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":196806,"spinlock_wait_cycles":127659904,"tablet-delete.queue_time_us":255619}
I20260924 19:57:33.694896 18231 tablet_replica.cc:333] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:33.695019 18224 raft_consensus.cc:359] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.695709 18224 raft_consensus.cc:385] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:33.695623 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 151247d6b5254cd884c58db9311ac04a (table default.Table_c8e27940040c495780f92f92431541cb [id=47a10be86ad441a7961c0f34fed46040]) successfully deleted
I20260924 19:57:33.695955 18224 raft_consensus.cc:740] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:33.696549 18224 consensus_queue.cc:260] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.697938 18224 ts_tablet_manager.cc:1434] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260924 19:57:33.596987 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "954178b8cc1e434085337f66b0e85672" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:33.699180 17823 raft_consensus.cc:2468] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:33.646546 17822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a524bad1c4040e58ad84fcf5a4fc47d" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:33.699771 17822 raft_consensus.cc:2468] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:33.698799 18224 tablet_bootstrap.cc:492] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:33.703840 17610 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f169c47d711d4566b2004cdfd0de725d has no consensus state
W20260924 19:57:33.704559 17606 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dbb82914df65416aaf47db8e507afb40 consensus state has no leader
I20260924 19:57:33.705642 18224 tablet_bootstrap.cc:654] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:33.716495 18231 raft_consensus.cc:2243] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:33.720595 18231 pending_rounds.cc:70] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: Trying to abort 1 pending ops.
I20260924 19:57:33.720897 18231 pending_rounds.cc:77] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986267563761664 op_type: NO_OP noop_request { }
I20260924 19:57:33.721298 18231 raft_consensus.cc:2889] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:33.725070 18231 raft_consensus.cc:2272] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:33.725555 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 6bd442f16492495f84bb8363b9bade97 (DEFAULT_TABLE table=default.Table_f40037fb3c91495e83bfe25056d5012f [id=738602c27af64fc2a1b0f1b1ddae9e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:33.726590 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd442f16492495f84bb8363b9bade97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.728335 17796 tablet_service.cc:1511] Processing CreateTablet for tablet e0246d83189d4ba5b75fbd8390f517f6 (DEFAULT_TABLE table=default.Table_202c25f88e1a485c9ca6677d9cf53e75 [id=221fc6b9725747ada3ed73dc569052cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:33.729305 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0246d83189d4ba5b75fbd8390f517f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.732558 17795 tablet_service.cc:1511] Processing CreateTablet for tablet 07f86f24d9d14d05898e1c0f69428850 (DEFAULT_TABLE table=default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:33.733578 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f86f24d9d14d05898e1c0f69428850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.734760 18320 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.2.190:46309,127.17.2.189:43689,127.17.2.188:38761
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.190:46309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.2.190
--webserver_port=37347
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:57:33.735787 17800 tablet_service.cc:1511] Processing CreateTablet for tablet affce4de8e0b412aa3b4bf19f9be7b6e (DEFAULT_TABLE table=default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:33.736536 18320 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:57:33.736665 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affce4de8e0b412aa3b4bf19f9be7b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.738778 18320 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260924 19:57:33.738343 17424 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 48226.3us
I20260924 19:57:33.752339 18224 tablet_bootstrap.cc:492] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:33.752756 18224 ts_tablet_manager.cc:1403] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.005s
I20260924 19:57:33.769101 18231 ts_tablet_manager.cc:1916] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:33.775576 18578 consensus_queue.cc:1048] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:33.777189 18224 raft_consensus.cc:359] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.777796 18224 raft_consensus.cc:385] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:33.778086 18224 raft_consensus.cc:740] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:33.778673 18224 consensus_queue.cc:260] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.781021 18224 ts_tablet_manager.cc:1434] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.028s user 0.003s sys 0.002s
I20260924 19:57:33.781985 18224 tablet_bootstrap.cc:492] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:33.796066 18224 tablet_bootstrap.cc:654] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:33.803195 18088 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 102) took 1252 ms. Trace:
I20260924 19:57:33.701634 17906 leader_election.cc:304] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
W20260924 19:57:33.804683 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 103140us
I20260924 19:57:33.805917 18426 raft_consensus.cc:2804] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:33.806565 18426 raft_consensus.cc:493] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:33.806890 17943 tablet_service.cc:1511] Processing CreateTablet for tablet e0246d83189d4ba5b75fbd8390f517f6 (DEFAULT_TABLE table=default.Table_202c25f88e1a485c9ca6677d9cf53e75 [id=221fc6b9725747ada3ed73dc569052cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:33.808145 17943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0246d83189d4ba5b75fbd8390f517f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.807297 18426 raft_consensus.cc:3060] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:33.758487 18492 raft_consensus.cc:2272] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:33.758612 18118 raft_consensus.cc:2468] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:33.811546 17946 tablet_service.cc:1511] Processing CreateTablet for tablet 6bd442f16492495f84bb8363b9bade97 (DEFAULT_TABLE table=default.Table_f40037fb3c91495e83bfe25056d5012f [id=738602c27af64fc2a1b0f1b1ddae9e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:33.812578 18224 tablet_bootstrap.cc:492] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:33.812604 17946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd442f16492495f84bb8363b9bade97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.813002 18224 ts_tablet_manager.cc:1403] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.001s
I20260924 19:57:33.803632 18088 rpcz_store.cc:276] 0924 19:57:32.550594 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.627109 (+ 76515us) service_pool.cc:224] Handling call
0924 19:57:33.001445 (+374336us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:33.009007 (+ 7562us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:33.377565 (+368558us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:33.802618 (+425053us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:33.803108 (+ 490us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:33.813822 18561 raft_consensus.cc:2706] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopped
I20260924 19:57:33.815433 18224 raft_consensus.cc:359] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.816068 18224 raft_consensus.cc:385] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:33.816329 18224 raft_consensus.cc:740] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:33.820307 17906 leader_election.cc:304] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:33.820441 17942 tablet_service.cc:1511] Processing CreateTablet for tablet 07f86f24d9d14d05898e1c0f69428850 (DEFAULT_TABLE table=default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:33.821159 18413 raft_consensus.cc:2804] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:33.820927 18578 consensus_queue.cc:1048] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:33.821488 18413 raft_consensus.cc:493] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:33.821790 18413 raft_consensus.cc:3060] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:33.821872 17942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f86f24d9d14d05898e1c0f69428850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.823921 17947 tablet_service.cc:1511] Processing CreateTablet for tablet affce4de8e0b412aa3b4bf19f9be7b6e (DEFAULT_TABLE table=default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:33.825155 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affce4de8e0b412aa3b4bf19f9be7b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:33.825920 18224 consensus_queue.cc:260] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.827864 18224 ts_tablet_manager.cc:1434] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260924 19:57:33.827788 17753 leader_election.cc:304] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:33.828680 18224 tablet_bootstrap.cc:492] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:33.828739 18578 raft_consensus.cc:2804] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:33.829195 18578 raft_consensus.cc:697] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:33.829947 18578 consensus_queue.cc:237] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.835820 18504 consensus_queue.cc:1048] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:33.840353 18413 raft_consensus.cc:515] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.844565 18220 tablet_bootstrap.cc:492] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:33.844374 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a524bad1c4040e58ad84fcf5a4fc47d" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:33.844990 17823 raft_consensus.cc:3060] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:33.850766 18224 tablet_bootstrap.cc:654] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:33.865252 17823 raft_consensus.cc:2468] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:33.865499 18224 tablet_bootstrap.cc:492] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:33.867129 18224 ts_tablet_manager.cc:1403] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.000s
I20260924 19:57:33.869760 18224 raft_consensus.cc:359] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.870517 18224 raft_consensus.cc:385] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:33.870816 18224 raft_consensus.cc:740] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:33.876328 18224 consensus_queue.cc:260] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.881372 18224 ts_tablet_manager.cc:1434] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20260924 19:57:33.885049 18224 tablet_bootstrap.cc:492] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:33.893353 18224 tablet_bootstrap.cc:654] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:33.852818 18050 consensus_peers.cc:597] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32466931871c4c5d8e9f95a11ffe26be. This is attempt 1: this message will repeat every 5th retry.
W20260924 19:57:33.896724 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 72748.9us
I20260924 19:57:33.898622 18223 ts_tablet_manager.cc:1434] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.340s user 0.006s sys 0.005s
I20260924 19:57:33.902993 18223 tablet_bootstrap.cc:492] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:33.917042 17615 master_service.cc:960] ReplaceTablet: received request to replace tablet 5f994e0401df4fc19bc5abb10f08bc6e from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:33.921845 18413 leader_election.cc:290] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:33.911404 17906 leader_election.cc:304] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:33.932150 18413 raft_consensus.cc:2804] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:33.933405 18413 raft_consensus.cc:697] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:33.935055 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_081e10d31dc74797abba37099d8a75fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:33.935652 18413 consensus_queue.cc:237] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:33.936864 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_081e10d31dc74797abba37099d8a75fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:33.941869 18116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "201ea0020cb3430eac85e80b8c06fb07" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:33.942523 18116 raft_consensus.cc:3060] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:33.947472 18504 raft_consensus.cc:493] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eaf64bfb64aa42948c9bc1411ce4ce35)
I20260924 19:57:33.947952 18504 raft_consensus.cc:515] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.949963 18504 leader_election.cc:290] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:33.950079 17602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20260924 19:57:33.950860 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ee43b6dc064dc7b320691d52e57ea7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:33.988276 18540 raft_consensus.cc:493] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:33.988741 18540 raft_consensus.cc:515] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:33.990790 18540 leader_election.cc:290] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:33.993158 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24f84032bca24a7ca9da65f0bbceb0d5" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:33.995014 17753 leader_election.cc:343] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:33.997529 18220 tablet_bootstrap.cc:654] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:34.000705 18118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24f84032bca24a7ca9da65f0bbceb0d5" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:34.004874 17753 leader_election.cc:343] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.005306 17753 leader_election.cc:304] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:34.006609 18540 raft_consensus.cc:2749] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:34.010654 18540 raft_consensus.cc:493] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:34.011199 18540 raft_consensus.cc:515] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.013341 18540 leader_election.cc:290] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:34.017066 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a765abd702644f079bddffb3fc4238ef" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:34.019498 17753 leader_election.cc:343] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.023531 18578 raft_consensus.cc:493] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:34.024130 17607 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet ef3f82de70bb4328a15a044fe0e500bf
I20260924 19:57:34.025687 17607 catalog_manager.cc:6934] ReplaceTablet: tablet ef3f82de70bb4328a15a044fe0e500bf deleted and replaced by tablet 57008266b80345e0a3f8e60bd7e2d62b
W20260924 19:57:34.025620 17602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.190:46309, user_credentials={real_user=slave}} blocked reactor thread for 75560.3us
I20260924 19:57:34.031459 18348 ts_tablet_manager.cc:1929] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:34.031963 18348 log.cc:1199] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/f482b53a4a2d44f6b3431c588cd605ce
I20260924 19:57:34.032814 18348 ts_tablet_manager.cc:1950] T f482b53a4a2d44f6b3431c588cd605ce P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:34.032387 18540 raft_consensus.cc:493] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:34.034051 18540 raft_consensus.cc:515] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.037289 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 5f994e0401df4fc19bc5abb10f08bc6e (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]). Expected version 1 got 0
I20260924 19:57:34.037902 18540 leader_election.cc:290] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:34.037941 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet c884f12860164f12bbcd0d2f87067013 (table default.Table_5b3a0213ccbb4622a13a68e24d12af4e [id=54f05b78c3324978b2635be4bd969f66]). Expected version 1 got 0
I20260924 19:57:34.038621 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 97a1b8ccd35a43cea7a71d705fad1494 (table default.Table_850e5158d78d4850b87445dac75ef7a9 [id=f768ce157a7b407d9f1e55a5b9d0720a]). Expected version 1 got 0
I20260924 19:57:34.032581 18578 raft_consensus.cc:515] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.043113 17946 tablet_service.cc:1558] Processing DeleteTablet for tablet ef3f82de70bb4328a15a044fe0e500bf with delete_type TABLET_DATA_DELETED (replaced by tablet 57008266b80345e0a3f8e60bd7e2d62b) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:34.044338 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet ef3f82de70bb4328a15a044fe0e500bf with delete_type TABLET_DATA_DELETED (replaced by tablet 57008266b80345e0a3f8e60bd7e2d62b) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:34.046159 17753 leader_election.cc:343] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: a765abd702644f079bddffb3fc4238ef
I20260924 19:57:34.046545 17753 leader_election.cc:304] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:34.047267 18573 raft_consensus.cc:2749] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:34.049443 17609 catalog_manager.cc:5838] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
W20260924 19:57:33.988129 18111 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.012s sys 0.008s
I20260924 19:57:34.054821 18492 ts_tablet_manager.cc:1916] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:34.054943 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1684aaf9d984649bff8c4b0b12044b9" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:34.057560 17753 leader_election.cc:343] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:33.996497 18109 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.014s sys 0.016s
I20260924 19:57:34.050823 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_56ddb56503844349ba42c68ebb67a694 [id=522c4df3169f4d2ba1cbe15df69d40b3]). Expected version 1 got 0
W20260924 19:57:34.053733 18111 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.670s user 0.013s sys 0.008s
I20260924 19:57:34.064363 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 349c6e9413bb41268d0c41d549203e2a (table default.Table_41c56bb86db34f16aba6e94147cde1d2 [id=b1d9f8b4039b4d19bfc493355e0d07cc]). Expected version 1 got 0
I20260924 19:57:34.083447 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1684aaf9d984649bff8c4b0b12044b9" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:34.060205 18109 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s user 0.014s sys 0.016s
I20260924 19:57:34.088761 18107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "954178b8cc1e434085337f66b0e85672" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:34.089362 18107 raft_consensus.cc:2468] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:34.090515 18116 raft_consensus.cc:2468] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
W20260924 19:57:34.091055 17753 leader_election.cc:343] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.092672 17753 leader_election.cc:304] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:34.094177 17753 consensus_peers.cc:597] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f482b53a4a2d44f6b3431c588cd605ce. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:34.095594 18085 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 106) took 1362 ms. Trace:
I20260924 19:57:34.096030 18085 rpcz_store.cc:276] 0924 19:57:32.733592 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.735701 (+ 2109us) service_pool.cc:224] Handling call
0924 19:57:33.009896 (+274195us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:33.020001 (+ 10105us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:33.804651 (+784650us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.043865 (+239214us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:34.095511 (+ 51646us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24394}
I20260924 19:57:34.101328 18107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a524bad1c4040e58ad84fcf5a4fc47d" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:34.102837 18084 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 108) took 1364 ms. Trace:
I20260924 19:57:34.103224 18084 rpcz_store.cc:276] 0924 19:57:32.738145 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.744386 (+ 6241us) service_pool.cc:224] Handling call
0924 19:57:32.965285 (+220899us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606e7c 00007fb83c606a3b 00007fb84554fd19 00007fb84478ba70 00007fb844870e61 00007fb84484e0f3 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82 00007fb842d86ab1
0924 19:57:33.051826 (+ 86541us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:33.065816 (+ 13990us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:33.768747 (+702931us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.065079 (+296332us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:34.102781 (+ 37702us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17527,"spinlock_wait_cycles":6505216}
I20260924 19:57:34.104708 18090 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 100) took 1680 ms. Trace:
I20260924 19:57:34.105150 18090 rpcz_store.cc:276] 0924 19:57:32.424653 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.437354 (+ 12701us) service_pool.cc:224] Handling call
0924 19:57:32.812378 (+375024us) spinlock_profiling.cc:243] Waited 21.5 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c607182 00007fb845578ec9 00007fb84478b918 00007fb84484e1b0 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82 00007fb842d86ab1 00007fb842d86a2c
0924 19:57:33.031844 (+219466us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:33.046327 (+ 14483us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:33.794546 (+748219us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.086028 (+291482us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:34.104639 (+ 18611us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4756,"spinlock_wait_cycles":49412736}
I20260924 19:57:34.107195 18086 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 109) took 1336 ms. Trace:
I20260924 19:57:34.107630 18086 rpcz_store.cc:276] 0924 19:57:32.770663 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.881492 (+110829us) service_pool.cc:224] Handling call
0924 19:57:33.049157 (+167665us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:33.050940 (+ 1783us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:33.763372 (+712432us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.093852 (+330480us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:34.100511 (+ 6659us) spinlock_profiling.cc:243] Waited 5.8 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c607182 00007fb845578ec9 00007fb84478b918 00007fb844851fa3 00007fb84484dd3f 00007fb84484e4fd 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82
0924 19:57:34.107092 (+ 6581us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6287,"spinlock_wait_cycles":13334400}
I20260924 19:57:34.114854 18540 raft_consensus.cc:2749] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:34.124413 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet f482b53a4a2d44f6b3431c588cd605ce (table default.Table_b814c73599bb4d9fae83012912129997 [id=abad09090ec04805bf6260c67fd479c1]) successfully deleted
I20260924 19:57:34.124792 18348 tablet_replica.cc:333] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:34.125748 18348 raft_consensus.cc:2243] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:34.126575 18348 raft_consensus.cc:2272] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:34.122416 18426 raft_consensus.cc:515] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.145242 18426 leader_election.cc:290] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:34.150106 18107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a524bad1c4040e58ad84fcf5a4fc47d" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:34.150534 18111 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.17.2.129:42273 (request call id 45) took 1465 ms. Trace:
I20260924 19:57:34.156255 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "954178b8cc1e434085337f66b0e85672" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:34.156394 17608 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_4d4391b7ade9428a8ed817a952ac87f9" } new_table_name: "default.Table_818936b597264e41853b635f6232f509" modify_external_catalogs: true
I20260924 19:57:34.156850 17819 raft_consensus.cc:3060] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:34.163532 17819 raft_consensus.cc:2468] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:34.165061 18348 ts_tablet_manager.cc:1916] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:34.165189 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_12bd04903f454058b69aa57c470e9ead"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:34.167212 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12bd04903f454058b69aa57c470e9ead in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:34.175009 17612 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_01931a7dc8a94582ae8c31c6078b66c5" } new_table_name: "default.Table_3b871d019f0d40c4b916ef34227f5730" modify_external_catalogs: true
I20260924 19:57:34.179224 17906 leader_election.cc:304] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:34.179391 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:34.181072 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:34.185967 18631 raft_consensus.cc:2804] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:34.186416 18224 tablet_bootstrap.cc:492] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:34.186524 18631 raft_consensus.cc:697] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:34.186874 18224 ts_tablet_manager.cc:1403] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.305s user 0.054s sys 0.042s
I20260924 19:57:34.187637 18631 consensus_queue.cc:237] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.190838 18224 raft_consensus.cc:359] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.191567 18224 raft_consensus.cc:385] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:34.191860 18224 raft_consensus.cc:740] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:34.192512 18224 consensus_queue.cc:260] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.196231 18224 ts_tablet_manager.cc:1434] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260924 19:57:34.199672 18578 leader_election.cc:290] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:34.200249 18224 tablet_bootstrap.cc:492] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:34.206015 18644 tablet.cc:1751] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260924 19:57:34.208709 17608 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92c0daa7d67e4aa390e334c669adda76 has no consensus state
I20260924 19:57:34.215034 18224 tablet_bootstrap.cc:654] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:34.222273 18578 raft_consensus.cc:493] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eaf64bfb64aa42948c9bc1411ce4ce35)
I20260924 19:57:34.222884 18578 raft_consensus.cc:515] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:34.237489 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 136097us
I20260924 19:57:34.241307 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd6e656a653a43d1843a91fee813237d" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:34.241665 18631 consensus_queue.cc:1048] T 7c7b8422753a4746bde71fcb25409949 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:34.237402 18644 tablet.cc:1751] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.246858 18224 tablet_bootstrap.cc:492] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:34.247352 18224 ts_tablet_manager.cc:1403] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.047s user 0.013s sys 0.000s
W20260924 19:57:34.248451 17753 leader_election.cc:343] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.248413 18645 tablet.cc:1751] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.179042 18231 ts_tablet_manager.cc:1929] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260924 19:57:34.255259 17905 leader_election.cc:343] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.258320 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet a765abd702644f079bddffb3fc4238ef with delete_type TABLET_DATA_DELETED (replaced by tablet f169c47d711d4566b2004cdfd0de725d) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:34.250984 18224 raft_consensus.cc:359] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.259097 18224 raft_consensus.cc:385] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:34.259486 18224 raft_consensus.cc:740] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:34.260478 18578 leader_election.cc:290] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:34.260298 18224 consensus_queue.cc:260] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.265085 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6943a365da464e83cedc2b9711ca89" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:34.268275 18224 ts_tablet_manager.cc:1434] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
W20260924 19:57:34.269837 17599 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92c0daa7d67e4aa390e334c669adda76 has no consensus state
I20260924 19:57:34.271210 18220 tablet_bootstrap.cc:492] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:34.271665 18220 ts_tablet_manager.cc:1403] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.427s user 0.014s sys 0.000s
I20260924 19:57:34.274461 18224 tablet_bootstrap.cc:492] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:34.275677 18095 tablet_service.cc:1511] Processing CreateTablet for tablet 6bd442f16492495f84bb8363b9bade97 (DEFAULT_TABLE table=default.Table_f40037fb3c91495e83bfe25056d5012f [id=738602c27af64fc2a1b0f1b1ddae9e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.274250 18220 raft_consensus.cc:359] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.276386 18220 raft_consensus.cc:385] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:34.277141 18220 raft_consensus.cc:740] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:34.278105 18220 consensus_queue.cc:260] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.284013 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 07f86f24d9d14d05898e1c0f69428850 (DEFAULT_TABLE table=default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.284508 18087 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 105) took 1551 ms. Trace:
I20260924 19:57:34.284845 18087 rpcz_store.cc:276] 0924 19:57:32.733284 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.737026 (+ 3742us) service_pool.cc:224] Handling call
0924 19:57:33.023457 (+286431us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:33.043389 (+ 19932us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:33.790272 (+746883us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.099216 (+308944us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:34.120103 (+ 20887us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c607182 00007fb845578ec9 00007fb84478b918 00007fb844851fa3 00007fb84484dd3f 00007fb84484e4fd 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82
0924 19:57:34.284453 (+164350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29991680}
I20260924 19:57:34.281143 18220 ts_tablet_manager.cc:1434] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260924 19:57:34.286078 18082 tablet_service.cc:1511] Processing CreateTablet for tablet affce4de8e0b412aa3b4bf19f9be7b6e (DEFAULT_TABLE table=default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.286695 18220 tablet_bootstrap.cc:492] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
W20260924 19:57:34.287930 17905 leader_election.cc:343] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.259197 18231 log.cc:1199] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/efd7339d0fee436299a7eac02a82addd
W20260924 19:57:34.149783 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 563172us
I20260924 19:57:34.290557 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c6943a365da464e83cedc2b9711ca89" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:34.291749 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f86f24d9d14d05898e1c0f69428850. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:34.292968 18646 tablet.cc:1751] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.295240 18087 tablet_service.cc:1558] Processing DeleteTablet for tablet ef3f82de70bb4328a15a044fe0e500bf with delete_type TABLET_DATA_DELETED (replaced by tablet 57008266b80345e0a3f8e60bd7e2d62b) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:34.295729 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd442f16492495f84bb8363b9bade97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:34.296056 18116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "83ee43b6dc064dc7b320691d52e57ea7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:34.301440 18645 tablet.cc:1751] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.276767 18089 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 101) took 1734 ms. Trace:
I20260924 19:57:34.295645 18224 tablet_bootstrap.cc:654] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:34.298557 18231 ts_tablet_manager.cc:1950] T efd7339d0fee436299a7eac02a82addd P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:34.313097 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 105) took 1760 ms. Trace:
I20260924 19:57:34.308926 18098 tablet_service.cc:1511] Processing CreateTablet for tablet e0246d83189d4ba5b75fbd8390f517f6 (DEFAULT_TABLE table=default.Table_202c25f88e1a485c9ca6677d9cf53e75 [id=221fc6b9725747ada3ed73dc569052cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.316530 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0246d83189d4ba5b75fbd8390f517f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:34.317453 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet affce4de8e0b412aa3b4bf19f9be7b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:34.323647 18092 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 98) took 2067 ms. Trace:
I20260924 19:57:34.303318 18641 raft_consensus.cc:493] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:34.324401 18641 raft_consensus.cc:515] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.324132 18092 rpcz_store.cc:276] 0924 19:57:32.255865 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.264963 (+ 9098us) service_pool.cc:224] Handling call
0924 19:57:32.699123 (+434160us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:32.758619 (+ 59496us) spinlock_profiling.cc:243] Waited 908 us on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c6072ee 00007fb845578f09 00007fb84478b9e1 00007fb84484e2f9 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82 00007fb842d86ab1 00007fb842d86a2c
0924 19:57:32.758684 (+ 65us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:33.745834 (+987150us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.086500 (+340666us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:34.287554 (+201054us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c607182 00007fb845578ec9 00007fb84478b918 00007fb844851fa3 00007fb84484dd3f 00007fb84484e4fd 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82
0924 19:57:34.323546 (+ 35992us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1004,"spinlock_wait_cycles":48773632}
I20260924 19:57:34.327142 18641 leader_election.cc:290] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:34.334942 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_415dca00ed274e59961ddf45e42de178"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:34.337062 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_415dca00ed274e59961ddf45e42de178 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:34.302677 17753 leader_election.cc:343] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: fd6e656a653a43d1843a91fee813237d
I20260924 19:57:34.348387 17753 leader_election.cc:304] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:34.349608 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 46968.5us
I20260924 19:57:34.323838 18231 rpcz_store.cc:276] 0924 19:57:32.552290 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.552435 (+ 145us) service_pool.cc:224] Handling call
0924 19:57:33.694744 (+1142309us) ts_tablet_manager.cc:1151] Deleting tablet efd7339d0fee436299a7eac02a82addd
0924 19:57:33.694789 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.078575 (+383786us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.178904 (+100329us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.313010 (+134106us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5695,"tablet-delete.queue_time_us":1115021}
I20260924 19:57:34.352124 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet efd7339d0fee436299a7eac02a82addd (table default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]) successfully deleted
I20260924 19:57:34.353263 18231 tablet_replica.cc:333] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:34.354806 18223 tablet_bootstrap.cc:654] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:34.343169 18578 raft_consensus.cc:493] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:34.356650 18578 raft_consensus.cc:515] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.359262 18578 leader_election.cc:290] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:34.360203 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_5aeda90fd8e3425a8c49a239533690c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:34.361908 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5aeda90fd8e3425a8c49a239533690c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:34.354084 18641 raft_consensus.cc:2749] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:34.366460 18224 tablet_bootstrap.cc:492] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:34.366505 18231 raft_consensus.cc:2243] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:34.371805 18224 ts_tablet_manager.cc:1403] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.098s user 0.012s sys 0.029s
I20260924 19:57:34.372252 18231 raft_consensus.cc:2272] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:34.374274 18224 raft_consensus.cc:359] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.374989 18224 raft_consensus.cc:385] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:34.375326 18224 raft_consensus.cc:740] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:34.377669 18107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e68378312c334fc28dafb71bf6386be8" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:34.375995 18224 consensus_queue.cc:260] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.383005 18231 ts_tablet_manager.cc:1916] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:34.386993 18224 ts_tablet_manager.cc:1434] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260924 19:57:34.388093 18224 tablet_bootstrap.cc:492] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:34.389729 17753 leader_election.cc:304] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:34.395320 18641 raft_consensus.cc:2749] T 0c6943a365da464e83cedc2b9711ca89 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:57:34.399907 17753 leader_election.cc:343] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.401958 18642 tablet.cc:1751] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.404277 17610 master_service.cc:960] ReplaceTablet: received request to replace tablet ef3f82de70bb4328a15a044fe0e500bf from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:34.379127 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b7f2dd8ba2a4cb190e9ff92a542bacb" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:34.407598 18348 ts_tablet_manager.cc:1929] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:34.408056 18348 log.cc:1199] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/916877c7540a4f6685064fc2807cb140
I20260924 19:57:34.408793 18348 ts_tablet_manager.cc:1950] T 916877c7540a4f6685064fc2807cb140 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
W20260924 19:57:34.410844 17753 leader_election.cc:343] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.415704 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "954178b8cc1e434085337f66b0e85672" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:34.416253 18112 raft_consensus.cc:3060] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:34.419857 18652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:57:34.400107 18224 tablet_bootstrap.cc:654] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:34.150893 18111 rpcz_store.cc:276] 0924 19:57:32.685121 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.685246 (+ 125us) service_pool.cc:224] Handling call
0924 19:57:32.783403 (+ 98157us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c606a02 00007fb84554fd19 00007fb84478ba70 00007fb8448546db 00007fb844854761 00007fb844816570 00007fb844800b69 00007fb844800612 00007fb842e7cd34
0924 19:57:32.783603 (+ 200us) raft_consensus.cc:1471] Updating replica for 1 ops
0924 19:57:32.979064 (+195461us) raft_consensus.cc:1534] Early marking committed up to index 0
0924 19:57:32.979094 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0924 19:57:32.988721 (+ 9627us) log.cc:844] Serialized 26 byte log entry
0924 19:57:33.204545 (+215824us) raft_consensus.cc:1645] Marking committed up to 1
0924 19:57:34.079134 (+874589us) log.cc:844] Serialized 14 byte log entry
0924 19:57:34.098344 (+ 19210us) raft_consensus.cc:1697] Filling consensus response to leader.
0924 19:57:34.120586 (+ 22242us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0924 19:57:34.140185 (+ 19599us) raft_consensus.cc:1685] finished
0924 19:57:34.150291 (+ 10106us) raft_consensus.cc:1691] UpdateReplicas() finished
0924 19:57:34.150441 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":759290,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":33,"raft.queue_time_us":15945,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":279,"spinlock_wait_cycles":33314048,"thread_start_us":797628,"threads_started":2,"wal-append.queue_time_us":279216}
I20260924 19:57:34.315395 18089 rpcz_store.cc:276] 0924 19:57:32.513915 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.543717 (+ 29802us) service_pool.cc:224] Handling call
0924 19:57:32.964367 (+420650us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606e7c 00007fb83c606a3b 00007fb84554fd19 00007fb84478ba70 00007fb844870e61 00007fb84484e0f3 00007fb8447fc640 00007fb842d86bf4 00007fb842d86b82 00007fb842d86ab1
0924 19:57:33.063925 (+ 99558us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:33.067967 (+ 4042us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:33.779679 (+711712us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.058119 (+278440us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:34.247908 (+189789us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6425344}
I20260924 19:57:34.428268 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b7f2dd8ba2a4cb190e9ff92a542bacb" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:34.431550 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e68378312c334fc28dafb71bf6386be8" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:34.432066 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 109) took 1191 ms. Trace:
I20260924 19:57:34.432442 18348 rpcz_store.cc:276] 0924 19:57:33.241093 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.241188 (+ 95us) service_pool.cc:224] Handling call
0924 19:57:34.124594 (+883406us) ts_tablet_manager.cc:1151] Deleting tablet 916877c7540a4f6685064fc2807cb140
0924 19:57:34.124647 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.340753 (+216106us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.407479 (+ 66726us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.432006 (+ 24527us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":882889}
I20260924 19:57:34.433873 18348 tablet_replica.cc:333] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:34.434052 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 916877c7540a4f6685064fc2807cb140 (table default.Table_f80e5f6313d9425f9721192ea215cb74 [id=a0bdd622a33241118711b9828d48e1d3]) successfully deleted
I20260924 19:57:34.437106 18220 tablet_bootstrap.cc:654] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:34.441352 18224 tablet_bootstrap.cc:492] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:34.441776 18224 ts_tablet_manager.cc:1403] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.003s
I20260924 19:57:34.444293 18224 raft_consensus.cc:359] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.444990 18224 raft_consensus.cc:385] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:34.445925 18348 raft_consensus.cc:2243] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:34.450315 18231 ts_tablet_manager.cc:1929] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:34.456101 18231 log.cc:1199] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/085b8088134d47478f0ce06a6a9f8572
I20260924 19:57:34.456996 18231 ts_tablet_manager.cc:1950] T 085b8088134d47478f0ce06a6a9f8572 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:34.455646 18224 raft_consensus.cc:740] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:34.457778 18224 consensus_queue.cc:260] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.461079 18224 ts_tablet_manager.cc:1434] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.019s user 0.004s sys 0.003s
I20260924 19:57:34.464010 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 106) took 1909 ms. Trace:
W20260924 19:57:34.474589 17612 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edc6c16578e5463a9d44c25fd6c538a9 consensus state has no leader
I20260924 19:57:34.475035 18231 rpcz_store.cc:276] 0924 19:57:32.555093 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.555244 (+ 151us) service_pool.cc:224] Handling call
0924 19:57:34.353050 (+1797806us) ts_tablet_manager.cc:1151] Deleting tablet 085b8088134d47478f0ce06a6a9f8572
0924 19:57:34.353111 (+ 61us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.418540 (+ 65429us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.450186 (+ 31646us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.463940 (+ 13754us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773138}
I20260924 19:57:34.465348 18224 tablet_bootstrap.cc:492] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:34.476615 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 085b8088134d47478f0ce06a6a9f8572 (table default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]) successfully deleted
I20260924 19:57:34.483349 18224 tablet_bootstrap.cc:654] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:34.484218 18231 tablet_replica.cc:333] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
W20260924 19:57:34.491199 17753 leader_election.cc:343] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.491652 17753 leader_election.cc:304] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:34.492559 18641 raft_consensus.cc:2749] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:34.492782 17753 leader_election.cc:343] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.493219 17753 leader_election.cc:304] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:34.496685 18641 raft_consensus.cc:2749] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:34.498608 18646 tablet.cc:1751] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.476048 17612 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b2d71701938e43c2bb30b8594cb20cf0" } new_table_name: "default.Table_22092e7254a9428e81176f410a135394" modify_external_catalogs: true
I20260924 19:57:34.517050 18231 raft_consensus.cc:2243] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:34.530972 17905 leader_election.cc:304] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:34.537226 18224 tablet_bootstrap.cc:492] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:34.537770 18224 ts_tablet_manager.cc:1403] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.073s user 0.015s sys 0.013s
I20260924 19:57:34.548403 17615 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 5f994e0401df4fc19bc5abb10f08bc6e
I20260924 19:57:34.549837 17615 catalog_manager.cc:6934] ReplaceTablet: tablet 5f994e0401df4fc19bc5abb10f08bc6e deleted and replaced by tablet e1137318a9fa4a0781f718dc34c4ecbe
I20260924 19:57:34.555769 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_6b68af48dd4140dda7d1854a3cf2269f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:34.557433 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b68af48dd4140dda7d1854a3cf2269f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:34.564170 18659 tablet.cc:1751] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.565301 18348 raft_consensus.cc:2272] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:34.545640 18051 consensus_peers.cc:597] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efd7339d0fee436299a7eac02a82addd. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:34.566231 18089 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f994e0401df4fc19bc5abb10f08bc6e with delete_type TABLET_DATA_DELETED (replaced by tablet e1137318a9fa4a0781f718dc34c4ecbe) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:34.576538 18636 raft_consensus.cc:2749] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:34.575255 18631 consensus_queue.cc:1048] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260924 19:57:34.584545 18348 ts_tablet_manager.cc:1916] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:34.586081 18231 raft_consensus.cc:2272] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:34.587168 18578 raft_consensus.cc:493] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:34.587291 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f994e0401df4fc19bc5abb10f08bc6e with delete_type TABLET_DATA_DELETED (replaced by tablet e1137318a9fa4a0781f718dc34c4ecbe) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:34.587603 18578 raft_consensus.cc:515] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.588172 18231 ts_tablet_manager.cc:1916] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:34.587812 18645 tablet.cc:1751] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.589664 18578 leader_election.cc:290] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:34.592375 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 94570.3us
I20260924 19:57:34.592185 18224 raft_consensus.cc:359] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:34.592893 18224 raft_consensus.cc:385] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:34.593176 18224 raft_consensus.cc:740] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:34.593861 18224 consensus_queue.cc:260] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:34.596580 18224 ts_tablet_manager.cc:1434] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.058s user 0.004s sys 0.001s
I20260924 19:57:34.597481 18224 tablet_bootstrap.cc:492] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:34.599143 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f994e0401df4fc19bc5abb10f08bc6e with delete_type TABLET_DATA_DELETED (replaced by tablet e1137318a9fa4a0781f718dc34c4ecbe) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:34.566206 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 122380us
I20260924 19:57:34.600634 18636 raft_consensus.cc:493] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:34.601462 18504 raft_consensus.cc:493] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:34.605628 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cf62a73c49245198d65011875148dc3" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:34.604435 18636 raft_consensus.cc:515] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.612022 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd434e38c9364f55bd18383149015028" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:34.613178 18224 tablet_bootstrap.cc:654] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:34.621847 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cf62a73c49245198d65011875148dc3" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:34.606607 18504 raft_consensus.cc:515] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.632539 18231 ts_tablet_manager.cc:1929] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:34.632958 18231 log.cc:1199] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/916877c7540a4f6685064fc2807cb140
I20260924 19:57:34.635394 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24f84032bca24a7ca9da65f0bbceb0d5" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:34.636065 17823 raft_consensus.cc:2468] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:34.633749 18231 ts_tablet_manager.cc:1950] T 916877c7540a4f6685064fc2807cb140 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:34.637946 17906 leader_election.cc:304] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:34.638674 18413 raft_consensus.cc:2804] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:34.639039 18413 raft_consensus.cc:493] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:34.639428 18413 raft_consensus.cc:3060] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:34.640195 18224 tablet_bootstrap.cc:492] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:34.641548 18224 ts_tablet_manager.cc:1403] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.000s
I20260924 19:57:34.648389 18413 raft_consensus.cc:515] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.668006 18578 raft_consensus.cc:493] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:34.668499 18224 raft_consensus.cc:359] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:34.668623 18578 raft_consensus.cc:515] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.670686 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 107) took 2105 ms. Trace:
I20260924 19:57:34.671012 18231 rpcz_store.cc:276] 0924 19:57:32.565054 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.565233 (+ 179us) service_pool.cc:224] Handling call
0924 19:57:34.484047 (+1918814us) ts_tablet_manager.cc:1151] Deleting tablet 916877c7540a4f6685064fc2807cb140
0924 19:57:34.484100 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.616021 (+131921us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.632397 (+ 16376us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.670618 (+ 38221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":126280,"tablet-delete.queue_time_us":1903957}
I20260924 19:57:34.672406 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 108) took 2105 ms. Trace:
I20260924 19:57:34.669207 18224 raft_consensus.cc:385] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:34.673274 18224 raft_consensus.cc:740] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:34.672761 18231 rpcz_store.cc:276] 0924 19:57:32.566651 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.567568 (+ 917us) service_pool.cc:224] Handling call
0924 19:57:34.672113 (+2104545us) ts_tablet_manager.cc:1151] Deleting tablet a384c9a0bb65429ebb97fe0807c78773
0924 19:57:34.672177 (+ 64us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.672341 (+ 164us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2090646}
I20260924 19:57:34.674219 18231 tablet_replica.cc:333] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:34.674077 18224 consensus_queue.cc:260] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:34.675078 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 916877c7540a4f6685064fc2807cb140 (table default.Table_f80e5f6313d9425f9721192ea215cb74 [id=a0bdd622a33241118711b9828d48e1d3]) successfully deleted
I20260924 19:57:34.675292 18231 raft_consensus.cc:2243] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:34.675877 18231 raft_consensus.cc:2272] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:34.675904 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet a384c9a0bb65429ebb97fe0807c78773 because the tablet was not found. No further retry: Not found: Tablet not found: a384c9a0bb65429ebb97fe0807c78773
I20260924 19:57:34.672436 18578 leader_election.cc:290] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:34.678117 18231 ts_tablet_manager.cc:1916] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:34.678737 18224 ts_tablet_manager.cc:1434] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.037s user 0.006s sys 0.000s
I20260924 19:57:34.680294 18224 tablet_bootstrap.cc:492] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:34.693465 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd434e38c9364f55bd18383149015028" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:34.694259 18113 raft_consensus.cc:2393] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate c4f6e3a7b06148918d230c77ab0c8202 in this term.
I20260924 19:57:34.699110 18224 tablet_bootstrap.cc:654] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:34.672724 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 71530.4us
I20260924 19:57:34.708945 18107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c861e93ad9f40ba9ad010021e43bc2b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:34.711829 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24f84032bca24a7ca9da65f0bbceb0d5" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:34.712473 17823 raft_consensus.cc:3060] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:34.682782 18112 raft_consensus.cc:2468] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:34.684099 18636 leader_election.cc:290] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:34.691360 18504 leader_election.cc:290] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:34.702550 17905 leader_election.cc:304] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:34.709041 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c861e93ad9f40ba9ad010021e43bc2b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:34.722622 18231 ts_tablet_manager.cc:1929] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:34.723590 18504 raft_consensus.cc:3060] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:34.726009 18492 ts_tablet_manager.cc:1929] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:34.726606 18492 log.cc:1199] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/f38188eccc344246b72a8755ec5201ed
I20260924 19:57:34.727496 18492 ts_tablet_manager.cc:1950] T f38188eccc344246b72a8755ec5201ed P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:34.728348 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 91) took 3214 ms. Trace:
I20260924 19:57:34.728799 18492 rpcz_store.cc:276] 0924 19:57:31.514147 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:31.514281 (+ 134us) service_pool.cc:224] Handling call
0924 19:57:33.268522 (+1754241us) ts_tablet_manager.cc:1151] Deleting tablet f38188eccc344246b72a8755ec5201ed
0924 19:57:33.288196 (+ 19674us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.410037 (+1121841us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.725825 (+315788us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.728262 (+ 2437us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":362040,"tablet-delete.queue_time_us":1731421}
I20260924 19:57:34.732060 18231 log.cc:1199] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/32466931871c4c5d8e9f95a11ffe26be
I20260924 19:57:34.731649 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24f84032bca24a7ca9da65f0bbceb0d5" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:34.733250 18231 ts_tablet_manager.cc:1950] T 32466931871c4c5d8e9f95a11ffe26be P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
W20260924 19:57:34.736096 17753 leader_election.cc:343] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.737963 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet f38188eccc344246b72a8755ec5201ed (table default.Table_37482fb26ba144619bb66861dceaab3b [id=06b8b073d6ae475c8125613b89463d3c]) successfully deleted
I20260924 19:57:34.743990 18220 tablet_bootstrap.cc:492] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:34.744581 18220 ts_tablet_manager.cc:1403] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.458s user 0.018s sys 0.002s
W20260924 19:57:34.749159 17753 leader_election.cc:343] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:34.750022 17753 leader_election.cc:343] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.750502 17753 leader_election.cc:304] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:34.719110 17823 raft_consensus.cc:2468] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:34.746908 18668 tablet.cc:1751] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.752490 17906 leader_election.cc:304] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:34.753043 18223 tablet_bootstrap.cc:492] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:34.753283 18636 raft_consensus.cc:2804] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:34.753474 18223 ts_tablet_manager.cc:1403] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.851s user 0.017s sys 0.005s
I20260924 19:57:34.755589 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 109) took 2169 ms. Trace:
I20260924 19:57:34.756075 18231 rpcz_store.cc:276] 0924 19:57:32.586661 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.587017 (+ 356us) service_pool.cc:224] Handling call
0924 19:57:34.674024 (+2087007us) ts_tablet_manager.cc:1151] Deleting tablet 32466931871c4c5d8e9f95a11ffe26be
0924 19:57:34.674080 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.715239 (+ 41159us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.722497 (+ 7258us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.755491 (+ 32994us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13036,"tablet-delete.queue_time_us":2086402}
I20260924 19:57:34.757170 18636 raft_consensus.cc:697] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:34.757869 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 32466931871c4c5d8e9f95a11ffe26be (table default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06]) successfully deleted
I20260924 19:57:34.758176 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 113) took 1768 ms. Trace:
I20260924 19:57:34.758894 18636 consensus_queue.cc:237] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.752717 18553 raft_consensus.cc:2749] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:34.764025 17905 leader_election.cc:343] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.766467 18578 raft_consensus.cc:2749] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:34.768064 18224 tablet_bootstrap.cc:492] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:34.768558 18224 ts_tablet_manager.cc:1403] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.088s user 0.016s sys 0.008s
W20260924 19:57:34.770222 17753 leader_election.cc:343] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:34.770608 17753 leader_election.cc:304] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:34.778206 18578 raft_consensus.cc:2749] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:34.779726 18224 raft_consensus.cc:359] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:34.785696 18224 raft_consensus.cc:385] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:34.786693 18224 raft_consensus.cc:740] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:34.788170 18224 consensus_queue.cc:260] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:34.790583 18413 leader_election.cc:290] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:34.766388 18117 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.17.2.129:42273 (request call id 56) took 1320 ms. Trace:
W20260924 19:57:34.790923 18051 consensus_peers.cc:597] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32466931871c4c5d8e9f95a11ffe26be. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:34.758563 18231 rpcz_store.cc:276] 0924 19:57:32.989868 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.039434 (+ 49566us) service_pool.cc:224] Handling call
0924 19:57:34.757705 (+1718271us) ts_tablet_manager.cc:1151] Deleting tablet d970cc92a1134a98a613d198a319cb01
0924 19:57:34.757772 (+ 67us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.758103 (+ 331us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1717542}
I20260924 19:57:34.791008 18117 rpcz_store.cc:276] 0924 19:57:33.445978 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.463283 (+ 17305us) service_pool.cc:224] Handling call
0924 19:57:33.493737 (+ 30454us) raft_consensus.cc:1471] Updating replica for 2 ops
0924 19:57:33.494635 (+ 898us) raft_consensus.cc:1534] Early marking committed up to index 0
0924 19:57:33.494673 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0924 19:57:33.495243 (+ 570us) alter_schema_op.cc:128] START. Timestamp: 1790279853245730
0924 19:57:33.979644 (+484401us) log.cc:844] Serialized 199 byte log entry
0924 19:57:34.379998 (+400354us) raft_consensus.cc:1645] Marking committed up to 2
0924 19:57:34.442876 (+ 62878us) log.cc:844] Serialized 14 byte log entry
0924 19:57:34.746624 (+303748us) raft_consensus.cc:1697] Filling consensus response to leader.
0924 19:57:34.753753 (+ 7129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0924 19:57:34.754255 (+ 502us) raft_consensus.cc:1685] finished
0924 19:57:34.758844 (+ 4589us) raft_consensus.cc:1691] UpdateReplicas() finished
0924 19:57:34.766191 (+ 7347us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:34.069584 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0924 19:57:34.070047 (+ 463us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0924 19:57:34.070084 (+ 37us) alter_schema_op.cc:66] Acquired schema lock
0924 19:57:34.070163 (+ 79us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0924 19:57:34.648946 (+578783us) op_driver.cc:464] REPLICATION: finished
0924 19:57:34.745258 (+ 96312us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0924 19:57:34.764731 (+ 19473us) log.cc:844] Serialized 5 byte log entry
Metrics: {"mutex_wait_us":3185,"prepare.queue_time_us":13446,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":9208,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":323,"thread_start_us":449583,"threads_started":2,"wal-append.queue_time_us":301572,"child_traces":[["op",{"apply.queue_time_us":94164,"prepare.queue_time_us":573927,"prepare.run_cpu_time_us":650,"prepare.run_wall_time_us":650,"replication_time_us":1145002,"thread_start_us":554227,"threads_started":2}]]}
I20260924 19:57:34.798300 18224 ts_tablet_manager.cc:1434] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20260924 19:57:34.802769 18659 tablet.cc:1751] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.810282 17607 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_7c382a08aec546da8c1aa720956036c7" } new_table_name: "default.Table_a22d035e711649318fbcf76ad552a553" modify_external_catalogs: true
W20260924 19:57:34.825927 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet d970cc92a1134a98a613d198a319cb01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d970cc92a1134a98a613d198a319cb01 already in progress: creating tablet
I20260924 19:57:34.826826 17795 tablet_service.cc:1511] Processing CreateTablet for tablet 2e6d926ff7c24901a15b0c7b2239fbad (DEFAULT_TABLE table=default.Table_4df831f35ae74b2f86e7e1b653b69867 [id=65f397bcbc164d0caa6cecae1bab74da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.827860 18671 tablet.cc:1751] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.730974 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 103) took 2064 ms. Trace:
I20260924 19:57:34.831310 18492 rpcz_store.cc:276] 0924 19:57:32.698183 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.698441 (+ 258us) service_pool.cc:224] Handling call
0924 19:57:34.730461 (+2032020us) ts_tablet_manager.cc:1151] Deleting tablet 41bdb846ca19425b882434df0167479e
0924 19:57:34.730524 (+ 63us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.730899 (+ 375us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1997632}
I20260924 19:57:34.832417 18492 tablet_replica.cc:333] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:34.833498 18492 raft_consensus.cc:2243] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:34.833983 18492 raft_consensus.cc:2272] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:34.835888 18492 ts_tablet_manager.cc:1916] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:34.806744 18223 raft_consensus.cc:359] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.840955 18223 raft_consensus.cc:385] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:34.841243 18223 raft_consensus.cc:740] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:34.843623 18223 consensus_queue.cc:260] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:34.845710 18223 ts_tablet_manager.cc:1434] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.092s user 0.007s sys 0.001s
I20260924 19:57:34.846637 18223 tablet_bootstrap.cc:492] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:34.849773 18659 tablet.cc:1751] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:34.858652 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_fbe470367ca848268ffa88adbee30894"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:34.860251 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbe470367ca848268ffa88adbee30894 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:34.875880 17606 master_service.cc:960] ReplaceTablet: received request to replace tablet 07f86f24d9d14d05898e1c0f69428850 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:34.878273 18223 tablet_bootstrap.cc:654] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:34.883188 17796 tablet_service.cc:1511] Processing CreateTablet for tablet 5df3dcf4ddcf4efa9aa047da56dfb7a7 (DEFAULT_TABLE table=default.Table_a82c256e36e448c296c86a30176f4135 [id=757ccfb2d39f4907a2580b365d283ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.888356 17947 tablet_service.cc:1511] Processing CreateTablet for tablet 2e6d926ff7c24901a15b0c7b2239fbad (DEFAULT_TABLE table=default.Table_4df831f35ae74b2f86e7e1b653b69867 [id=65f397bcbc164d0caa6cecae1bab74da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.889402 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6d926ff7c24901a15b0c7b2239fbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:34.896044 17946 tablet_service.cc:1511] Processing CreateTablet for tablet 5df3dcf4ddcf4efa9aa047da56dfb7a7 (DEFAULT_TABLE table=default.Table_a82c256e36e448c296c86a30176f4135 [id=757ccfb2d39f4907a2580b365d283ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.897217 17946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df3dcf4ddcf4efa9aa047da56dfb7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:34.900707 17943 tablet_service.cc:1511] Processing CreateTablet for tablet 274f3decdc7e42c0917bf28151c4da71 (DEFAULT_TABLE table=default.Table_954c770a550c49599e38ee3850e4aea1 [id=c4ad22b552204e2d8209b275e57eb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.901773 17943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274f3decdc7e42c0917bf28151c4da71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:34.953581 18492 ts_tablet_manager.cc:1929] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:34.954022 18492 log.cc:1199] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/9c2440e1cade4327a63d6234e3adcd64
I20260924 19:57:34.954036 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:34.954682 18492 ts_tablet_manager.cc:1950] T 9c2440e1cade4327a63d6234e3adcd64 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:34.969609 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 2e6d926ff7c24901a15b0c7b2239fbad (DEFAULT_TABLE table=default.Table_4df831f35ae74b2f86e7e1b653b69867 [id=65f397bcbc164d0caa6cecae1bab74da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.970700 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6d926ff7c24901a15b0c7b2239fbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:34.974365 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 274f3decdc7e42c0917bf28151c4da71 (DEFAULT_TABLE table=default.Table_954c770a550c49599e38ee3850e4aea1 [id=c4ad22b552204e2d8209b275e57eb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.975894 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 5df3dcf4ddcf4efa9aa047da56dfb7a7 (DEFAULT_TABLE table=default.Table_a82c256e36e448c296c86a30176f4135 [id=757ccfb2d39f4907a2580b365d283ace]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.976912 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df3dcf4ddcf4efa9aa047da56dfb7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:34.978914 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 92c0daa7d67e4aa390e334c669adda76 (DEFAULT_TABLE table=default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.979940 18095 tablet_service.cc:1511] Processing CreateTablet for tablet 274f3decdc7e42c0917bf28151c4da71 (DEFAULT_TABLE table=default.Table_954c770a550c49599e38ee3850e4aea1 [id=c4ad22b552204e2d8209b275e57eb9c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:34.980998 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274f3decdc7e42c0917bf28151c4da71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:34.983057 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 104) took 2250 ms. Trace:
I20260924 19:57:34.983418 18492 rpcz_store.cc:276] 0924 19:57:32.728916 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.733773 (+ 4857us) service_pool.cc:224] Handling call
0924 19:57:34.832252 (+2098479us) ts_tablet_manager.cc:1151] Deleting tablet 9c2440e1cade4327a63d6234e3adcd64
0924 19:57:34.832302 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.925944 (+ 93642us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.953434 (+ 27490us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:34.979051 (+ 25617us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2096889}
I20260924 19:57:34.984689 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 107) took 2246 ms. Trace:
I20260924 19:57:34.984997 18492 rpcz_store.cc:276] 0924 19:57:32.737828 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:32.741280 (+ 3452us) service_pool.cc:224] Handling call
0924 19:57:34.984413 (+2243133us) ts_tablet_manager.cc:1151] Deleting tablet a8cc802bf87f4d739175f93203478af8
0924 19:57:34.984458 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.984635 (+ 177us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2226085}
I20260924 19:57:34.991665 18231 tablet_replica.cc:333] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:34.992542 18231 raft_consensus.cc:2243] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:34.993410 18231 raft_consensus.cc:2272] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:34.996104 18231 ts_tablet_manager.cc:1916] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:35.000626 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 112) took 1959 ms. Trace:
I20260924 19:57:35.000779 18224 tablet_bootstrap.cc:492] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:35.000990 18492 rpcz_store.cc:276] 0924 19:57:33.043793 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.045680 (+ 1887us) service_pool.cc:224] Handling call
0924 19:57:34.985860 (+1940180us) ts_tablet_manager.cc:1151] Deleting tablet 151247d6b5254cd884c58db9311ac04a
0924 19:57:34.985910 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.000557 (+ 14647us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1938470}
I20260924 19:57:35.002046 18492 tablet_replica.cc:333] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:35.006649 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e6d926ff7c24901a15b0c7b2239fbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.011852 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df3dcf4ddcf4efa9aa047da56dfb7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.012452 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c0daa7d67e4aa390e334c669adda76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.002899 18492 raft_consensus.cc:2243] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:35.015444 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274f3decdc7e42c0917bf28151c4da71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.016009 18492 raft_consensus.cc:2272] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:35.018019 18492 ts_tablet_manager.cc:1916] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:35.018851 18224 tablet_bootstrap.cc:654] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:35.045555 18641 consensus_queue.cc:1048] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:35.049108 18641 raft_consensus.cc:493] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:35.069268 18578 raft_consensus.cc:493] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:35.069777 18578 raft_consensus.cc:515] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.071812 18578 leader_election.cc:290] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:35.079173 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41bdb846ca19425b882434df0167479e" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:35.082406 17753 leader_election.cc:343] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:35.084415 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41bdb846ca19425b882434df0167479e" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:35.086652 17753 leader_election.cc:343] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:35.087143 17753 leader_election.cc:304] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:35.089927 18641 raft_consensus.cc:515] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:35.092165 18641 leader_election.cc:290] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:35.092657 18641 raft_consensus.cc:2749] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:35.095045 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c08f10d2a364cb8a0a7ea40654c3fe4" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:35.096575 17753 leader_election.cc:343] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:35.100148 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c08f10d2a364cb8a0a7ea40654c3fe4" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:35.102633 17753 leader_election.cc:343] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:35.104400 17753 leader_election.cc:304] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:35.112771 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 97a1b8ccd35a43cea7a71d705fad1494 (table default.Table_850e5158d78d4850b87445dac75ef7a9 [id=f768ce157a7b407d9f1e55a5b9d0720a]). Expected version 1 got 0
I20260924 19:57:35.039739 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_ca51e47e4fc44d9ba5dd07cbeefa6e68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:35.114701 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca51e47e4fc44d9ba5dd07cbeefa6e68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:34.969944 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 41bdb846ca19425b882434df0167479e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41bdb846ca19425b882434df0167479e already in progress: creating tablet
W20260924 19:57:35.118292 17599 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 148383us
I20260924 19:57:35.121945 17942 tablet_service.cc:1511] Processing CreateTablet for tablet 92c0daa7d67e4aa390e334c669adda76 (DEFAULT_TABLE table=default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.123347 17942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c0daa7d67e4aa390e334c669adda76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.138286 18492 ts_tablet_manager.cc:1929] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:35.138682 18492 log.cc:1199] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/efd7339d0fee436299a7eac02a82addd
I20260924 19:57:35.152856 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 9c2440e1cade4327a63d6234e3adcd64 (table default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]) successfully deleted
I20260924 19:57:35.153076 18095 tablet_service.cc:1511] Processing CreateTablet for tablet 92c0daa7d67e4aa390e334c669adda76 (DEFAULT_TABLE table=default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:57:35.153962 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet a8cc802bf87f4d739175f93203478af8 because the tablet was not found. No further retry: Not found: Tablet not found: a8cc802bf87f4d739175f93203478af8
I20260924 19:57:35.154281 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92c0daa7d67e4aa390e334c669adda76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:34.750746 18220 raft_consensus.cc:359] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:35.154678 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 151247d6b5254cd884c58db9311ac04a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 151247d6b5254cd884c58db9311ac04a already in progress: creating tablet
I20260924 19:57:35.154812 18220 raft_consensus.cc:385] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:35.155098 18220 raft_consensus.cc:740] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:35.158365 18220 consensus_queue.cc:260] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.160754 18220 ts_tablet_manager.cc:1434] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.415s user 0.007s sys 0.000s
I20260924 19:57:35.161779 18220 tablet_bootstrap.cc:492] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:35.163622 18492 ts_tablet_manager.cc:1950] T efd7339d0fee436299a7eac02a82addd P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:35.164592 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 113) took 2112 ms. Trace:
I20260924 19:57:35.164964 18492 rpcz_store.cc:276] 0924 19:57:33.069326 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.070258 (+ 932us) service_pool.cc:224] Handling call
0924 19:57:35.001890 (+1931632us) ts_tablet_manager.cc:1151] Deleting tablet efd7339d0fee436299a7eac02a82addd
0924 19:57:35.001931 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.077530 (+ 75599us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.138154 (+ 60624us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.164527 (+ 26373us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1931109}
I20260924 19:57:35.166253 18492 tablet_replica.cc:333] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:35.178445 18231 ts_tablet_manager.cc:1929] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:35.164654 18504 raft_consensus.cc:2749] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:35.170854 18636 consensus_queue.cc:1048] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:35.174347 18348 ts_tablet_manager.cc:1929] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:35.183473 18348 log.cc:1199] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/085b8088134d47478f0ce06a6a9f8572
I20260924 19:57:35.184623 18348 ts_tablet_manager.cc:1950] T 085b8088134d47478f0ce06a6a9f8572 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:35.193626 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 1 got 0
I20260924 19:57:35.194782 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:35.203333 18223 tablet_bootstrap.cc:492] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:35.203768 18223 ts_tablet_manager.cc:1403] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.357s user 0.022s sys 0.023s
I20260924 19:57:35.205963 18223 raft_consensus.cc:359] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.206557 18223 raft_consensus.cc:385] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:35.206785 18223 raft_consensus.cc:740] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:35.207741 18223 consensus_queue.cc:260] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.209975 18223 ts_tablet_manager.cc:1434] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260924 19:57:35.167124 18492 raft_consensus.cc:2243] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:35.210814 18492 raft_consensus.cc:2272] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:35.213028 18492 ts_tablet_manager.cc:1916] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:35.165076 18641 raft_consensus.cc:2749] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:35.198683 18677 raft_consensus.cc:493] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:35.219066 17797 tablet_service.cc:1511] Processing CreateTablet for tablet ba46ac86b242407abd3e8dbf09f12efa (DEFAULT_TABLE table=default.Table_6115a46b8e7444e6899712d31f41390a [id=6e83504d58ee4c52b83c9109bb7bfa1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.223799 18683 consensus_queue.cc:1048] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260924 19:57:35.245906 18224 tablet_bootstrap.cc:492] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:35.242048 18677 raft_consensus.cc:515] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.246454 18224 ts_tablet_manager.cc:1403] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.246s user 0.019s sys 0.001s
I20260924 19:57:35.240074 18231 log.cc:1199] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/f482b53a4a2d44f6b3431c588cd605ce
I20260924 19:57:35.247790 18231 ts_tablet_manager.cc:1950] T f482b53a4a2d44f6b3431c588cd605ce P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:35.257158 18223 tablet_bootstrap.cc:492] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:35.257606 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 110) took 2016 ms. Trace:
I20260924 19:57:35.257987 18348 rpcz_store.cc:276] 0924 19:57:33.241432 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.241536 (+ 104us) service_pool.cc:224] Handling call
0924 19:57:34.433704 (+1192168us) ts_tablet_manager.cc:1151] Deleting tablet 085b8088134d47478f0ce06a6a9f8572
0924 19:57:34.433754 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:34.662462 (+228708us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.174208 (+511746us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.257542 (+ 83334us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7388,"tablet-delete.queue_time_us":1191647}
I20260924 19:57:35.262428 18223 tablet_bootstrap.cc:654] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:35.249485 18677 leader_election.cc:290] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:35.264495 18348 tablet_replica.cc:333] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:35.267460 18348 raft_consensus.cc:2243] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:35.268666 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 114) took 2169 ms. Trace:
I20260924 19:57:35.269963 18348 raft_consensus.cc:2272] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:35.272526 18231 rpcz_store.cc:276] 0924 19:57:33.099261 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.099387 (+ 126us) service_pool.cc:224] Handling call
0924 19:57:34.797020 (+1697633us) ts_tablet_manager.cc:1151] Deleting tablet f482b53a4a2d44f6b3431c588cd605ce
0924 19:57:34.991489 (+194469us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.143616 (+152127us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.178328 (+ 34712us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.268601 (+ 90273us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1696709}
I20260924 19:57:35.269537 18220 tablet_bootstrap.cc:654] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:35.265578 17943 tablet_service.cc:1511] Processing CreateTablet for tablet ba46ac86b242407abd3e8dbf09f12efa (DEFAULT_TABLE table=default.Table_6115a46b8e7444e6899712d31f41390a [id=6e83504d58ee4c52b83c9109bb7bfa1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.276274 17943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba46ac86b242407abd3e8dbf09f12efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.250334 18224 raft_consensus.cc:359] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:35.279374 18224 raft_consensus.cc:385] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:35.279696 18224 raft_consensus.cc:740] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:35.280388 18224 consensus_queue.cc:260] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:35.281682 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 117) took 1918 ms. Trace:
I20260924 19:57:35.272217 18348 ts_tablet_manager.cc:1916] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:35.285866 18231 rpcz_store.cc:276] 0924 19:57:33.363601 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.364542 (+ 941us) service_pool.cc:224] Handling call
0924 19:57:35.281431 (+1916889us) ts_tablet_manager.cc:1151] Deleting tablet 9c2440e1cade4327a63d6234e3adcd64
0924 19:57:35.281472 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.281627 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1916314}
I20260924 19:57:35.267984 17606 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 07f86f24d9d14d05898e1c0f69428850
I20260924 19:57:35.293331 18231 tablet_replica.cc:333] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:35.285429 18224 ts_tablet_manager.cc:1434] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20260924 19:57:35.293625 18223 tablet_bootstrap.cc:492] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:35.294200 18223 ts_tablet_manager.cc:1403] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.000s
I20260924 19:57:35.294764 18224 tablet_bootstrap.cc:492] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:35.295059 17606 catalog_manager.cc:6934] ReplaceTablet: tablet 07f86f24d9d14d05898e1c0f69428850 deleted and replaced by tablet 1d95d6e48173436c8c59130464413ea1
I20260924 19:57:35.296811 18223 raft_consensus.cc:359] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.297457 18223 raft_consensus.cc:385] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:35.297560 18492 ts_tablet_manager.cc:1929] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:35.297725 18223 raft_consensus.cc:740] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:35.297926 18492 log.cc:1199] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/916877c7540a4f6685064fc2807cb140
I20260924 19:57:35.298583 18492 ts_tablet_manager.cc:1950] T 916877c7540a4f6685064fc2807cb140 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:35.298367 18223 consensus_queue.cc:260] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.302138 18220 tablet_bootstrap.cc:492] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:35.302402 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_56ddb56503844349ba42c68ebb67a694 [id=522c4df3169f4d2ba1cbe15df69d40b3]). Expected version 1 got 0
I20260924 19:57:35.302646 18220 ts_tablet_manager.cc:1403] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.141s user 0.008s sys 0.003s
I20260924 19:57:35.305388 18220 raft_consensus.cc:359] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.308395 18220 raft_consensus.cc:385] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:35.309998 18220 raft_consensus.cc:740] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:35.313508 17615 catalog_manager.cc:5838] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:35.315263 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:35.316447 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24f84032bca24a7ca9da65f0bbceb0d5" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:35.316998 18113 raft_consensus.cc:3060] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:35.325263 18113 raft_consensus.cc:2468] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:35.326750 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 114) took 2185 ms. Trace:
I20260924 19:57:35.327097 18492 rpcz_store.cc:276] 0924 19:57:33.141587 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.144970 (+ 3383us) service_pool.cc:224] Handling call
0924 19:57:35.166081 (+2021111us) ts_tablet_manager.cc:1151] Deleting tablet 916877c7540a4f6685064fc2807cb140
0924 19:57:35.166125 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.245776 (+ 79651us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.297438 (+ 51662us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.326688 (+ 29250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18539,"tablet-delete.queue_time_us":2020550}
I20260924 19:57:35.328212 18492 tablet_replica.cc:333] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:35.328915 18492 raft_consensus.cc:2243] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:35.329635 18492 raft_consensus.cc:2272] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:35.327260 18220 consensus_queue.cc:260] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.334532 18492 ts_tablet_manager.cc:1916] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:35.340906 18223 ts_tablet_manager.cc:1434] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20260924 19:57:35.343957 18113 raft_consensus.cc:3060] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:35.344975 18223 tablet_bootstrap.cc:492] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:35.347679 18220 ts_tablet_manager.cc:1434] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20260924 19:57:35.356653 18413 consensus_queue.cc:1048] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:35.270852 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 085b8088134d47478f0ce06a6a9f8572 (table default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]) successfully deleted
I20260924 19:57:35.362392 18095 tablet_service.cc:1511] Processing CreateTablet for tablet ba46ac86b242407abd3e8dbf09f12efa (DEFAULT_TABLE table=default.Table_6115a46b8e7444e6899712d31f41390a [id=6e83504d58ee4c52b83c9109bb7bfa1b]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:57:35.362588 17599 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 91747.8us
I20260924 19:57:35.363355 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba46ac86b242407abd3e8dbf09f12efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.362385 18220 tablet_bootstrap.cc:492] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:35.364240 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet efd7339d0fee436299a7eac02a82addd (table default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]) successfully deleted
I20260924 19:57:35.365075 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 916877c7540a4f6685064fc2807cb140 (table default.Table_f80e5f6313d9425f9721192ea215cb74 [id=a0bdd622a33241118711b9828d48e1d3]) successfully deleted
I20260924 19:57:35.367010 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet 07f86f24d9d14d05898e1c0f69428850 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d95d6e48173436c8c59130464413ea1) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:35.370187 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:35.371284 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:35.383123 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_5208198cb618443dbec0f760012e4edd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:35.384799 18348 ts_tablet_manager.cc:1929] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260924 19:57:35.384955 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5208198cb618443dbec0f760012e4edd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:35.394721 18348 log.cc:1199] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/efd7339d0fee436299a7eac02a82addd
I20260924 19:57:35.396679 18348 ts_tablet_manager.cc:1950] T efd7339d0fee436299a7eac02a82addd P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:35.394232 18220 tablet_bootstrap.cc:654] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:35.399611 18223 tablet_bootstrap.cc:654] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:35.401706 17946 tablet_service.cc:1511] Processing CreateTablet for tablet df722b0bbfd04ce2943f0e5b8058c50f (DEFAULT_TABLE table=default.Table_bda24aa05e4c497385d8a40f8c9752b4 [id=0b7e96c71a074f838a519746d9c92c3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.413652 18492 ts_tablet_manager.cc:1929] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:35.414351 18492 log.cc:1199] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/32466931871c4c5d8e9f95a11ffe26be
I20260924 19:57:35.415932 18492 ts_tablet_manager.cc:1950] T 32466931871c4c5d8e9f95a11ffe26be P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:35.417169 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 1 got 0
I20260924 19:57:35.418223 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet 07f86f24d9d14d05898e1c0f69428850 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d95d6e48173436c8c59130464413ea1) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:35.428103 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet c1684aaf9d984649bff8c4b0b12044b9 (table default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe]). Expected version 1 got 0
W20260924 19:57:35.430672 17599 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92c0daa7d67e4aa390e334c669adda76 consensus state has no leader
I20260924 19:57:35.427119 17946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df722b0bbfd04ce2943f0e5b8058c50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.432251 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 108) took 2191 ms. Trace:
I20260924 19:57:35.433350 18348 rpcz_store.cc:276] 0924 19:57:33.240778 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.240886 (+ 108us) service_pool.cc:224] Handling call
0924 19:57:35.264345 (+2023459us) ts_tablet_manager.cc:1151] Deleting tablet efd7339d0fee436299a7eac02a82addd
0924 19:57:35.264389 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.329585 (+ 65196us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.384676 (+ 55091us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.432168 (+ 47492us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2022084}
I20260924 19:57:35.435304 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 111) took 2192 ms. Trace:
I20260924 19:57:35.439805 18348 rpcz_store.cc:276] 0924 19:57:33.242967 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.247042 (+ 4075us) service_pool.cc:224] Handling call
0924 19:57:35.435037 (+2187995us) ts_tablet_manager.cc:1151] Deleting tablet 9c2440e1cade4327a63d6234e3adcd64
0924 19:57:35.435081 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.435254 (+ 173us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2187363}
I20260924 19:57:35.442901 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 117) took 2137 ms. Trace:
I20260924 19:57:35.443341 18492 rpcz_store.cc:276] 0924 19:57:33.305186 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.305295 (+ 109us) service_pool.cc:224] Handling call
0924 19:57:35.328071 (+2022776us) ts_tablet_manager.cc:1151] Deleting tablet 32466931871c4c5d8e9f95a11ffe26be
0924 19:57:35.328112 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.359206 (+ 31094us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.413527 (+ 54321us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.442840 (+ 29313us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2018433}
I20260924 19:57:35.448036 18095 tablet_service.cc:1511] Processing CreateTablet for tablet df722b0bbfd04ce2943f0e5b8058c50f (DEFAULT_TABLE table=default.Table_bda24aa05e4c497385d8a40f8c9752b4 [id=0b7e96c71a074f838a519746d9c92c3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.450238 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df722b0bbfd04ce2943f0e5b8058c50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.454372 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet efd7339d0fee436299a7eac02a82addd (table default.Table_a07eb85445714c59bb9837bc7228144c [id=23c789fd31f64f30a41227eef0cb7690]) successfully deleted
I20260924 19:57:35.455749 18492 tablet_replica.cc:333] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:35.456498 18492 raft_consensus.cc:2243] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:35.457010 18492 raft_consensus.cc:2272] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:35.456832 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 44199d2655f14dcf97afe2180789aae4 (DEFAULT_TABLE table=default.Table_1b2f1b739e234953861c274a098c2e1c [id=f02f0018bf7346c6a0ba91d52909be98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.457854 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44199d2655f14dcf97afe2180789aae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.458833 18492 ts_tablet_manager.cc:1916] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:35.448398 18413 consensus_queue.cc:1048] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:35.473349 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 32466931871c4c5d8e9f95a11ffe26be (table default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06]) successfully deleted
I20260924 19:57:35.476223 17943 tablet_service.cc:1511] Processing CreateTablet for tablet f169c47d711d4566b2004cdfd0de725d (DEFAULT_TABLE table=default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.477433 17943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f169c47d711d4566b2004cdfd0de725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.451387 17942 tablet_service.cc:1511] Processing CreateTablet for tablet 44199d2655f14dcf97afe2180789aae4 (DEFAULT_TABLE table=default.Table_1b2f1b739e234953861c274a098c2e1c [id=f02f0018bf7346c6a0ba91d52909be98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.486332 17942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44199d2655f14dcf97afe2180789aae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.493645 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 112) took 2096 ms. Trace:
I20260924 19:57:35.506151 18492 ts_tablet_manager.cc:1929] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:35.506537 18492 log.cc:1199] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/f482b53a4a2d44f6b3431c588cd605ce
I20260924 19:57:35.513068 18223 tablet_bootstrap.cc:492] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:35.513653 18223 ts_tablet_manager.cc:1403] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.169s user 0.010s sys 0.005s
I20260924 19:57:35.514323 18348 rpcz_store.cc:276] 0924 19:57:33.397368 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.397531 (+ 163us) service_pool.cc:224] Handling call
0924 19:57:35.493295 (+2095764us) ts_tablet_manager.cc:1151] Deleting tablet a765abd702644f079bddffb3fc4238ef
0924 19:57:35.493338 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.493588 (+ 250us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2095069}
I20260924 19:57:35.515822 18348 tablet_replica.cc:333] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:35.516675 18348 raft_consensus.cc:2243] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:35.516371 18223 raft_consensus.cc:359] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.517299 18348 raft_consensus.cc:2272] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:35.517297 18223 raft_consensus.cc:385] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:35.517632 18223 raft_consensus.cc:740] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:35.521351 18636 raft_consensus.cc:493] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:35.521975 18636 raft_consensus.cc:515] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.525592 18636 leader_election.cc:290] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:35.525683 18492 ts_tablet_manager.cc:1950] T f482b53a4a2d44f6b3431c588cd605ce P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:35.518545 18223 consensus_queue.cc:260] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.534221 17610 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 97a1b8ccd35a43cea7a71d705fad1494 (table default.Table_850e5158d78d4850b87445dac75ef7a9 [id=f768ce157a7b407d9f1e55a5b9d0720a]). Expected version 1 got 0
I20260924 19:57:35.535632 17610 catalog_manager.cc:5838] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:35.537415 17610 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:35.539430 17610 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:35.550583 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 121) took 2214 ms. Trace:
I20260924 19:57:35.550911 18492 rpcz_store.cc:276] 0924 19:57:33.336217 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.339664 (+ 3447us) service_pool.cc:224] Handling call
0924 19:57:35.455604 (+2115940us) ts_tablet_manager.cc:1151] Deleting tablet f482b53a4a2d44f6b3431c588cd605ce
0924 19:57:35.455645 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.485674 (+ 30029us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.506027 (+ 20353us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.550517 (+ 44490us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2114262}
I20260924 19:57:35.552299 18492 tablet_replica.cc:333] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:35.553232 18492 raft_consensus.cc:2243] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:35.554605 18492 raft_consensus.cc:2272] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:35.560161 18223 ts_tablet_manager.cc:1434] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.046s user 0.009s sys 0.000s
I20260924 19:57:35.561184 18223 tablet_bootstrap.cc:492] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:35.563550 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872bc32b9e384c828a0699079b27530d" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:35.568393 18095 tablet_service.cc:1511] Processing CreateTablet for tablet f169c47d711d4566b2004cdfd0de725d (DEFAULT_TABLE table=default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.569483 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f169c47d711d4566b2004cdfd0de725d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.540756 17610 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_56ddb56503844349ba42c68ebb67a694 [id=522c4df3169f4d2ba1cbe15df69d40b3]). Expected version 1 got 0
I20260924 19:57:35.572397 17610 catalog_manager.cc:5838] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
W20260924 19:57:35.572988 17753 leader_election.cc:343] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:35.573679 17610 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 1 got 0
I20260924 19:57:35.574657 18223 tablet_bootstrap.cc:654] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:35.571483 17599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.189:43689, user_credentials={real_user=slave}} blocked reactor thread for 67635.8us
W20260924 19:57:35.576488 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 9c2440e1cade4327a63d6234e3adcd64 because the tablet was not found. No further retry: Not found: Tablet not found: 9c2440e1cade4327a63d6234e3adcd64
W20260924 19:57:35.577242 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet a765abd702644f079bddffb3fc4238ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a765abd702644f079bddffb3fc4238ef already in progress: creating tablet
I20260924 19:57:35.580274 18492 ts_tablet_manager.cc:1916] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:35.586347 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet f482b53a4a2d44f6b3431c588cd605ce (table default.Table_b814c73599bb4d9fae83012912129997 [id=abad09090ec04805bf6260c67fd479c1]) successfully deleted
W20260924 19:57:35.587085 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 9c2440e1cade4327a63d6234e3adcd64 because the tablet was not found. No further retry: Not found: Tablet not found: 9c2440e1cade4327a63d6234e3adcd64
I20260924 19:57:35.591277 17796 tablet_service.cc:1511] Processing CreateTablet for tablet df722b0bbfd04ce2943f0e5b8058c50f (DEFAULT_TABLE table=default.Table_bda24aa05e4c497385d8a40f8c9752b4 [id=0b7e96c71a074f838a519746d9c92c3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.592355 17795 tablet_service.cc:1511] Processing CreateTablet for tablet 44199d2655f14dcf97afe2180789aae4 (DEFAULT_TABLE table=default.Table_1b2f1b739e234953861c274a098c2e1c [id=f02f0018bf7346c6a0ba91d52909be98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.248132 17797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba46ac86b242407abd3e8dbf09f12efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.594515 17799 tablet_service.cc:1511] Processing CreateTablet for tablet f169c47d711d4566b2004cdfd0de725d (DEFAULT_TABLE table=default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:35.596026 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f169c47d711d4566b2004cdfd0de725d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:35.595953 17800 consensus_meta.cc:287] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202: Time spent flushing consensus metadata: real 0.526s user 0.004s sys 0.004s
I20260924 19:57:35.598809 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df722b0bbfd04ce2943f0e5b8058c50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.607491 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44199d2655f14dcf97afe2180789aae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:35.616578 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet 07f86f24d9d14d05898e1c0f69428850 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d95d6e48173436c8c59130464413ea1) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:35.617144 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0cf2df461598411f8ad9f3a51eafc6f7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:35.618320 17816 raft_consensus.cc:2468] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:35.672011 18224 tablet_bootstrap.cc:654] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:35.677660 18231 raft_consensus.cc:2243] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:35.678903 18231 raft_consensus.cc:2272] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:35.681880 18703 raft_consensus.cc:493] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:35.685537 18231 ts_tablet_manager.cc:1916] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:35.688230 18702 raft_consensus.cc:493] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:35.692653 18702 raft_consensus.cc:515] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:35.682431 18703 raft_consensus.cc:515] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:35.678861 18492 ts_tablet_manager.cc:1929] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:35.704648 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d970cc92a1134a98a613d198a319cb01" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:35.707587 18492 log.cc:1199] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/a2e4519aed7642678db3a518a89778de
I20260924 19:57:35.708634 18492 ts_tablet_manager.cc:1950] T a2e4519aed7642678db3a518a89778de P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:35.550073 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872bc32b9e384c828a0699079b27530d" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:35.710898 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 122) took 2374 ms. Trace:
W20260924 19:57:35.708990 17753 leader_election.cc:343] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:35.711241 18492 rpcz_store.cc:276] 0924 19:57:33.336597 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.341511 (+ 4914us) service_pool.cc:224] Handling call
0924 19:57:35.552102 (+2210591us) ts_tablet_manager.cc:1151] Deleting tablet a2e4519aed7642678db3a518a89778de
0924 19:57:35.552159 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.626812 (+ 74653us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.678686 (+ 51874us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.710836 (+ 32150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":477,"tablet-delete.queue_time_us":2205508}
I20260924 19:57:35.542146 18348 ts_tablet_manager.cc:1916] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:35.712623 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 120) took 2376 ms. Trace:
I20260924 19:57:35.712915 18492 rpcz_store.cc:276] 0924 19:57:33.335836 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.339319 (+ 3483us) service_pool.cc:224] Handling call
0924 19:57:35.712239 (+2372920us) ts_tablet_manager.cc:1151] Deleting tablet d970cc92a1134a98a613d198a319cb01
0924 19:57:35.712280 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.712517 (+ 237us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2346543}
I20260924 19:57:35.714027 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 123) took 1456 ms. Trace:
I20260924 19:57:35.714303 18492 rpcz_store.cc:276] 0924 19:57:34.258031 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.258166 (+ 135us) service_pool.cc:224] Handling call
0924 19:57:35.713694 (+1455528us) ts_tablet_manager.cc:1151] Deleting tablet a765abd702644f079bddffb3fc4238ef
0924 19:57:35.713733 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.713970 (+ 237us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1454826}
I20260924 19:57:35.715190 18492 tablet_replica.cc:333] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
W20260924 19:57:35.716364 17753 leader_election.cc:343] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:35.709468 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d970cc92a1134a98a613d198a319cb01" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:35.561955 18220 tablet_bootstrap.cc:492] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:35.718588 18220 ts_tablet_manager.cc:1403] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.358s user 0.011s sys 0.006s
I20260924 19:57:35.722030 18220 raft_consensus.cc:359] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.722790 18220 raft_consensus.cc:385] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:35.722818 18492 raft_consensus.cc:2243] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:35.723079 18220 raft_consensus.cc:740] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:35.725275 18492 raft_consensus.cc:2272] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:35.716792 17753 leader_election.cc:304] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:35.729559 18578 raft_consensus.cc:2749] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:35.731108 17753 leader_election.cc:343] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:35.733767 17753 leader_election.cc:304] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:35.712003 18703 leader_election.cc:290] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:35.735142 18683 raft_consensus.cc:2749] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:35.737046 18702 leader_election.cc:290] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:35.732687 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0feecd2fdb4409f86f6a5fbfbab3537" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:35.742175 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0feecd2fdb4409f86f6a5fbfbab3537" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:35.744057 17753 leader_election.cc:343] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:35.744427 18492 ts_tablet_manager.cc:1916] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:35.744916 17753 leader_election.cc:343] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:35.745365 17753 leader_election.cc:304] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:35.758713 18578 raft_consensus.cc:2749] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:35.787304 18492 ts_tablet_manager.cc:1929] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:35.791349 18492 log.cc:1199] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/ef3f82de70bb4328a15a044fe0e500bf
I20260924 19:57:35.800246 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0cf2df461598411f8ad9f3a51eafc6f7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:35.800994 18223 tablet_bootstrap.cc:492] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:35.801754 18223 ts_tablet_manager.cc:1403] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.241s user 0.031s sys 0.045s
I20260924 19:57:35.802059 18492 ts_tablet_manager.cc:1950] T ef3f82de70bb4328a15a044fe0e500bf P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:35.801721 18113 raft_consensus.cc:2468] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:35.804279 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 128) took 1516 ms. Trace:
I20260924 19:57:35.804499 18223 raft_consensus.cc:359] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.805171 18223 raft_consensus.cc:385] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:35.804654 18492 rpcz_store.cc:276] 0924 19:57:34.287409 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.287520 (+ 111us) service_pool.cc:224] Handling call
0924 19:57:35.715032 (+1427512us) ts_tablet_manager.cc:1151] Deleting tablet ef3f82de70bb4328a15a044fe0e500bf
0924 19:57:35.715069 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.768632 (+ 53563us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.787145 (+ 18513us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.804209 (+ 17064us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1419379}
I20260924 19:57:35.805478 18223 raft_consensus.cc:740] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:35.805182 18631 consensus_queue.cc:1048] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:35.806193 18492 tablet_replica.cc:333] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:35.807050 18492 raft_consensus.cc:2243] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:35.807628 18492 raft_consensus.cc:2272] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:35.808452 18223 consensus_queue.cc:260] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.809616 17906 leader_election.cc:304] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:35.810571 18223 ts_tablet_manager.cc:1434] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.008s user 0.000s sys 0.004s
I20260924 19:57:35.810514 18631 raft_consensus.cc:2804] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:35.810873 18631 raft_consensus.cc:493] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:35.811229 18631 raft_consensus.cc:3060] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:35.811502 18492 ts_tablet_manager.cc:1916] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:35.816977 18223 tablet_bootstrap.cc:492] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:35.818974 18631 raft_consensus.cc:515] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.822368 18223 tablet_bootstrap.cc:654] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:35.838797 18492 ts_tablet_manager.cc:1929] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:35.850206 18224 tablet_bootstrap.cc:492] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:35.845082 18223 tablet_bootstrap.cc:492] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:35.850798 18224 ts_tablet_manager.cc:1403] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.556s user 0.023s sys 0.005s
I20260924 19:57:35.850859 18492 log.cc:1199] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/5f994e0401df4fc19bc5abb10f08bc6e
I20260924 19:57:35.850960 18223 ts_tablet_manager.cc:1403] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.000s
I20260924 19:57:35.851984 18492 ts_tablet_manager.cc:1950] T 5f994e0401df4fc19bc5abb10f08bc6e P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:35.853744 18578 raft_consensus.cc:493] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:35.854177 18492 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 129) took 1293 ms. Trace:
I20260924 19:57:35.854206 18578 raft_consensus.cc:515] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:35.854341 18223 raft_consensus.cc:359] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.855013 18223 raft_consensus.cc:385] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:35.855319 18223 raft_consensus.cc:740] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:35.855953 18223 consensus_queue.cc:260] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.857821 18223 ts_tablet_manager.cc:1434] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260924 19:57:35.857724 18578 leader_election.cc:290] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:35.854525 18492 rpcz_store.cc:276] 0924 19:57:34.560639 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.566071 (+ 5432us) service_pool.cc:224] Handling call
0924 19:57:35.806037 (+1239966us) ts_tablet_manager.cc:1151] Deleting tablet 5f994e0401df4fc19bc5abb10f08bc6e
0924 19:57:35.806079 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:35.819069 (+ 12990us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.838649 (+ 19580us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.854116 (+ 15467us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1238791}
I20260924 19:57:35.869683 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbb82914df65416aaf47db8e507afb40" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:35.871199 18223 tablet_bootstrap.cc:492] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:35.873741 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbb82914df65416aaf47db8e507afb40" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:35.876617 18223 tablet_bootstrap.cc:654] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:35.881222 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet f482b53a4a2d44f6b3431c588cd605ce (table default.Table_b814c73599bb4d9fae83012912129997 [id=abad09090ec04805bf6260c67fd479c1]) successfully deleted
W20260924 19:57:35.888983 17753 leader_election.cc:343] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:35.890007 17753 consensus_peers.cc:597] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:35.891081 17614 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet fec9b6ecfb384b528cf71c4cfb2fd4cb
I20260924 19:57:35.895329 17614 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 167) took 2594 ms. Trace:
I20260924 19:57:35.902354 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:35.895995 17614 rpcz_store.cc:276] 0924 19:57:33.300362 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.300506 (+ 144us) service_pool.cc:224] Handling call
0924 19:57:33.378959 (+ 78453us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:33.379558 (+ 599us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:33.379681 (+ 123us) catalog_manager.cc:2863] Locking tablets
0924 19:57:33.722651 (+342970us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:34.022287 (+299636us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:34.022531 (+ 244us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:35.890902 (+1868371us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:35.895111 (+ 4209us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:33.733607 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:33.733959 (+ 352us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:33.733982 (+ 23us) write_op.cc:435] Acquired schema lock
0924 19:57:33.733995 (+ 13us) tablet.cc:662] Decoding operations
0924 19:57:33.734307 (+ 312us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:33.734339 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:33.734359 (+ 20us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:33.747559 (+ 13200us) tablet.cc:701] Row locks acquired
0924 19:57:33.747580 (+ 21us) write_op.cc:260] PREPARE: finished
0924 19:57:33.747711 (+ 131us) write_op.cc:270] Start()
0924 19:57:33.747796 (+ 85us) write_op.cc:276] Timestamp: P: 1790279853743377 usec, L: 0
0924 19:57:33.747810 (+ 14us) op_driver.cc:348] REPLICATION: starting
0924 19:57:33.748362 (+ 552us) log.cc:844] Serialized 728 byte log entry
0924 19:57:34.018449 (+270087us) op_driver.cc:464] REPLICATION: finished
0924 19:57:34.020957 (+ 2508us) write_op.cc:301] APPLY: starting
0924 19:57:34.021010 (+ 53us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:34.021121 (+ 111us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:34.021208 (+ 87us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:34.021464 (+ 256us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:34.021614 (+ 150us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:34.021665 (+ 51us) write_op.cc:312] APPLY: finished
0924 19:57:34.021836 (+ 171us) log.cc:844] Serialized 29 byte log entry
0924 19:57:34.021980 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:34.022051 (+ 71us) write_op.cc:454] Released schema lock
0924 19:57:34.022084 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:34.022313 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:34.022344 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2359,"apply.run_cpu_time_us":1333,"apply.run_wall_time_us":1333,"num_ops":2,"prepare.queue_time_us":7918,"prepare.run_cpu_time_us":2140,"prepare.run_wall_time_us":15934,"replication_time_us":270570}]]}
I20260924 19:57:35.909794 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_bb9775fce761443ca3a306e0cace741e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:35.912771 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb9775fce761443ca3a306e0cace741e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:35.939601 18220 consensus_queue.cc:260] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:35.942916 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0cf2df461598411f8ad9f3a51eafc6f7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:35.943610 17816 raft_consensus.cc:3060] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:35.953058 17816 raft_consensus.cc:2468] T 0cf2df461598411f8ad9f3a51eafc6f7 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
W20260924 19:57:36.035876 17753 leader_election.cc:343] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:36.037233 17753 leader_election.cc:304] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:35.983016 17906 leader_election.cc:304] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
W20260924 19:57:36.039991 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 57124.1us
I20260924 19:57:36.043654 18504 raft_consensus.cc:2804] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:36.045691 18413 raft_consensus.cc:493] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:36.047623 18413 raft_consensus.cc:515] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.073025 18223 tablet_bootstrap.cc:492] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:36.073467 18223 ts_tablet_manager.cc:1403] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.202s user 0.057s sys 0.027s
I20260924 19:57:36.077361 18348 ts_tablet_manager.cc:1929] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:36.077373 18223 raft_consensus.cc:359] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.078022 18223 raft_consensus.cc:385] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.078264 18223 raft_consensus.cc:740] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.078933 18223 consensus_queue.cc:260] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.079730 18348 log.cc:1199] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/ef3f82de70bb4328a15a044fe0e500bf
I20260924 19:57:36.082808 18348 ts_tablet_manager.cc:1950] T ef3f82de70bb4328a15a044fe0e500bf P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:36.085368 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 119) took 2042 ms. Trace:
I20260924 19:57:36.091594 18223 ts_tablet_manager.cc:1434] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.018s user 0.007s sys 0.001s
I20260924 19:57:36.091254 18348 rpcz_store.cc:276] 0924 19:57:34.042614 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.042966 (+ 352us) service_pool.cc:224] Handling call
0924 19:57:35.515673 (+1472707us) ts_tablet_manager.cc:1151] Deleting tablet ef3f82de70bb4328a15a044fe0e500bf
0924 19:57:35.515716 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:36.058454 (+542738us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.077211 (+ 18757us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.085298 (+ 8087us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471696}
I20260924 19:57:36.092715 18348 tablet_replica.cc:333] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:36.093843 18348 raft_consensus.cc:2243] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:36.094671 18348 raft_consensus.cc:2272] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:36.096590 18348 ts_tablet_manager.cc:1916] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:36.099424 18712 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:36.101930 18223 tablet_bootstrap.cc:492] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:36.101780 18712 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:36.108704 18561 raft_consensus.cc:493] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:36.109165 18561 raft_consensus.cc:515] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.112078 18561 leader_election.cc:290] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:36.107357 18223 tablet_bootstrap.cc:654] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:36.116515 17943 consensus_meta.cc:287] T f169c47d711d4566b2004cdfd0de725d P f74e6ed351284d7fb110e44336ce643a: Time spent flushing consensus metadata: real 0.631s user 0.010s sys 0.000s
I20260924 19:57:36.122946 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 215 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:36.123680 18708 raft_consensus.cc:493] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:36.124181 18708 raft_consensus.cc:515] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.125566 18712 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:36.125989 18348 ts_tablet_manager.cc:1929] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 19:57:36.132822 17522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260924 19:57:36.136412 18223 tablet_bootstrap.cc:492] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:36.135078 18231 ts_tablet_manager.cc:1929] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:36.136911 18223 ts_tablet_manager.cc:1403] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.008s
I20260924 19:57:36.137338 18231 log.cc:1199] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/a2e4519aed7642678db3a518a89778de
I20260924 19:57:36.137428 18701 raft_consensus.cc:2749] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:36.138396 18231 ts_tablet_manager.cc:1950] T a2e4519aed7642678db3a518a89778de P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:36.139804 18223 raft_consensus.cc:359] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:36.140446 18223 raft_consensus.cc:385] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.140690 18223 raft_consensus.cc:740] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.141427 18223 consensus_queue.cc:260] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:36.147802 18224 raft_consensus.cc:359] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:36.148545 18224 raft_consensus.cc:385] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.148844 18224 raft_consensus.cc:740] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.140636 18348 log.cc:1199] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/5f994e0401df4fc19bc5abb10f08bc6e
W20260924 19:57:36.149561 17522 leader_election.cc:336] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309): Network error: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111)
I20260924 19:57:36.150007 17522 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:36.149634 18224 consensus_queue.cc:260] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:36.150736 18712 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 19:57:36.151783 18223 ts_tablet_manager.cc:1434] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20260924 19:57:36.152328 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 116) took 2789 ms. Trace:
I20260924 19:57:36.150810 18348 ts_tablet_manager.cc:1950] T 5f994e0401df4fc19bc5abb10f08bc6e P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:36.155766 18231 rpcz_store.cc:276] 0924 19:57:33.363268 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.367710 (+ 4442us) service_pool.cc:224] Handling call
0924 19:57:35.293152 (+1925442us) ts_tablet_manager.cc:1151] Deleting tablet a2e4519aed7642678db3a518a89778de
0924 19:57:35.293205 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:36.126884 (+833679us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.134952 (+ 8068us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.152253 (+ 17301us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":382936,"tablet-delete.queue_time_us":1924937}
I20260924 19:57:36.156418 18710 consensus_queue.cc:1048] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:36.159762 18504 raft_consensus.cc:697] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:36.160135 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet a2e4519aed7642678db3a518a89778de (table default.Table_a4745efada6f4edea3f2d4bad1c3905e [id=993f4ef55cb44fb09a9c543e7cf1f2b8]) successfully deleted
I20260924 19:57:36.161072 18231 tablet_replica.cc:333] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:36.160719 18504 consensus_queue.cc:237] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.162252 18231 raft_consensus.cc:2243] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:36.161363 18348 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 121) took 1562 ms. Trace:
I20260924 19:57:36.162647 18348 rpcz_store.cc:276] 0924 19:57:34.598515 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.599003 (+ 488us) service_pool.cc:224] Handling call
0924 19:57:36.092556 (+1493553us) ts_tablet_manager.cc:1151] Deleting tablet 5f994e0401df4fc19bc5abb10f08bc6e
0924 19:57:36.092599 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:36.119063 (+ 26464us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.125857 (+ 6794us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.161304 (+ 35447us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1492961}
I20260924 19:57:36.164260 18231 raft_consensus.cc:2272] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:36.165819 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 113471us
I20260924 19:57:36.166579 18223 tablet_bootstrap.cc:492] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:36.168757 18231 ts_tablet_manager.cc:1916] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:36.171663 18223 tablet_bootstrap.cc:654] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:36.172564 17599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.189:43689, user_credentials={real_user=slave}} blocked reactor thread for 79234.7us
I20260924 19:57:36.174494 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:36.174544 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet ef3f82de70bb4328a15a044fe0e500bf (table default.Table_cca06980f109487a9ed8b8172950b60d [id=016f4039733144c985808b862eebcc9a]) successfully deleted
I20260924 19:57:36.175868 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 5f994e0401df4fc19bc5abb10f08bc6e (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]) successfully deleted
I20260924 19:57:36.076895 18220 ts_tablet_manager.cc:1434] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.358s user 0.007s sys 0.000s
I20260924 19:57:36.187706 18220 tablet_bootstrap.cc:492] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:36.202013 18223 tablet_bootstrap.cc:492] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:36.202411 18223 ts_tablet_manager.cc:1403] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.003s
W20260924 19:57:36.204319 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 168608us
I20260924 19:57:36.204753 18223 raft_consensus.cc:359] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.205385 18223 raft_consensus.cc:385] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.205612 18223 raft_consensus.cc:740] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.206187 18223 consensus_queue.cc:260] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.212277 18631 leader_election.cc:290] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:36.212945 18223 ts_tablet_manager.cc:1434] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260924 19:57:36.213794 18223 tablet_bootstrap.cc:492] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:36.216950 18050 leader_election.cc:343] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Not found: Tablet not found: 151247d6b5254cd884c58db9311ac04a
I20260924 19:57:36.224181 18708 leader_election.cc:290] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:36.228870 18231 ts_tablet_manager.cc:1929] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:36.229270 18231 log.cc:1199] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/a765abd702644f079bddffb3fc4238ef
I20260924 19:57:36.229910 18231 ts_tablet_manager.cc:1950] T a765abd702644f079bddffb3fc4238ef P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:36.234056 18413 leader_election.cc:290] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:36.235689 18223 tablet_bootstrap.cc:654] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:36.246629 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 118) took 2866 ms. Trace:
I20260924 19:57:36.247344 18231 rpcz_store.cc:276] 0924 19:57:33.380338 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.403254 (+ 22916us) service_pool.cc:224] Handling call
0924 19:57:36.160901 (+2757647us) ts_tablet_manager.cc:1151] Deleting tablet a765abd702644f079bddffb3fc4238ef
0924 19:57:36.160964 (+ 63us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:36.189326 (+ 28362us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.227100 (+ 37774us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.246522 (+ 19422us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2596,"tablet-delete.queue_time_us":2755919}
I20260924 19:57:36.248919 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet a765abd702644f079bddffb3fc4238ef (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]) successfully deleted
I20260924 19:57:36.249222 18231 tablet_replica.cc:333] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:36.250445 18231 raft_consensus.cc:2243] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:36.258975 18223 tablet_bootstrap.cc:492] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:36.259402 18223 ts_tablet_manager.cc:1403] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.011s
I20260924 19:57:36.261703 18223 raft_consensus.cc:359] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.262264 18223 raft_consensus.cc:385] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.262490 18223 raft_consensus.cc:740] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.263024 18223 consensus_queue.cc:260] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.264453 18231 raft_consensus.cc:2272] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:36.264904 18223 ts_tablet_manager.cc:1434] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 19:57:36.267220 18223 tablet_bootstrap.cc:492] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:36.268985 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b7f2dd8ba2a4cb190e9ff92a542bacb" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:36.269927 17816 raft_consensus.cc:2468] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:36.272137 17906 leader_election.cc:304] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:36.274583 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "910d61b8c3574302b5a328187226571e" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:36.275640 17815 raft_consensus.cc:2468] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:36.280582 18708 raft_consensus.cc:2804] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:36.281219 18708 raft_consensus.cc:493] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:36.281757 18708 raft_consensus.cc:3060] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.282145 18223 tablet_bootstrap.cc:654] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:36.285344 18224 ts_tablet_manager.cc:1434] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.434s user 0.008s sys 0.000s
I20260924 19:57:36.294212 17906 leader_election.cc:304] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:36.296057 18636 raft_consensus.cc:2804] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:36.296672 18636 raft_consensus.cc:493] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:36.297245 18636 raft_consensus.cc:3060] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.305837 18223 tablet_bootstrap.cc:492] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:36.306396 18223 ts_tablet_manager.cc:1403] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.012s
I20260924 19:57:36.309305 18223 raft_consensus.cc:359] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.309940 18223 raft_consensus.cc:385] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.310271 18223 raft_consensus.cc:740] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.311040 18223 consensus_queue.cc:260] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.313090 18223 ts_tablet_manager.cc:1434] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260924 19:57:36.314147 18223 tablet_bootstrap.cc:492] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:36.316345 18224 tablet_bootstrap.cc:492] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:36.322187 18223 tablet_bootstrap.cc:654] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:36.315305 18231 ts_tablet_manager.cc:1916] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:36.335860 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0cf2df461598411f8ad9f3a51eafc6f7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:36.336428 18117 raft_consensus.cc:3060] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.336937 18220 tablet_bootstrap.cc:654] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:36.336819 18561 raft_consensus.cc:493] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:36.337409 18561 raft_consensus.cc:515] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.340929 18561 leader_election.cc:290] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:36.343389 18117 raft_consensus.cc:2468] T 0cf2df461598411f8ad9f3a51eafc6f7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:36.344970 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "910d61b8c3574302b5a328187226571e" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:36.345533 18113 raft_consensus.cc:2468] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:36.345613 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b7f2dd8ba2a4cb190e9ff92a542bacb" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:36.346184 18117 raft_consensus.cc:2468] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:36.353902 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "910d61b8c3574302b5a328187226571e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:36.354585 17821 raft_consensus.cc:2468] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
W20260924 19:57:36.359766 18051 leader_election.cc:343] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 151247d6b5254cd884c58db9311ac04a
I20260924 19:57:36.360399 18051 leader_election.cc:304] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:36.363799 18561 raft_consensus.cc:2749] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:36.372717 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:36.372452 18051 leader_election.cc:304] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:36.374778 18220 tablet_bootstrap.cc:492] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:36.375348 18220 ts_tablet_manager.cc:1403] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.188s user 0.010s sys 0.002s
I20260924 19:57:36.376001 18561 raft_consensus.cc:2804] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:36.376756 18561 raft_consensus.cc:493] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:36.378548 18561 raft_consensus.cc:3060] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.390419 18631 consensus_queue.cc:1048] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:36.392130 18729 tablet_replica.cc:333] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:36.378767 18220 raft_consensus.cc:359] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.393436 18220 raft_consensus.cc:385] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.394649 18729 raft_consensus.cc:2243] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:36.395105 18220 raft_consensus.cc:740] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.399603 18220 consensus_queue.cc:260] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.400589 18729 raft_consensus.cc:2272] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:36.405465 17946 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51332 (request call id 128) took 1014 ms. Trace:
I20260924 19:57:36.406121 18224 tablet_bootstrap.cc:654] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:36.406347 18729 ts_tablet_manager.cc:1916] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:36.405916 17946 rpcz_store.cc:276] 0924 19:57:35.391527 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:35.391682 (+ 155us) service_pool.cc:224] Handling call
0924 19:57:35.419047 (+ 27365us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:35.419135 (+ 88us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:35.944541 (+525406us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.405011 (+460470us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:36.405402 (+ 391us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:36.407513 18220 ts_tablet_manager.cc:1434] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
W20260924 19:57:36.176479 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 07f86f24d9d14d05898e1c0f69428850 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07f86f24d9d14d05898e1c0f69428850 already in progress: creating tablet
W20260924 19:57:36.414698 17599 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 238219us
I20260924 19:57:36.416901 18220 tablet_bootstrap.cc:492] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
W20260924 19:57:36.419533 17599 catalog_manager.cc:5136] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): alter failed for tablet a384c9a0bb65429ebb97fe0807c78773 (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]),no further retry: Not found: Tablet not found: a384c9a0bb65429ebb97fe0807c78773
W20260924 19:57:36.420373 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fec9b6ecfb384b528cf71c4cfb2fd4cb already in progress: creating tablet
I20260924 19:57:36.428337 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet a2e4519aed7642678db3a518a89778de (table default.Table_a4745efada6f4edea3f2d4bad1c3905e [id=993f4ef55cb44fb09a9c543e7cf1f2b8]) successfully deleted
I20260924 19:57:36.430938 18220 tablet_bootstrap.cc:654] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:36.438489 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet d970cc92a1134a98a613d198a319cb01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d970cc92a1134a98a613d198a319cb01 already in progress: creating tablet
I20260924 19:57:36.413522 18708 raft_consensus.cc:515] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.459758 18708 leader_election.cc:290] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:36.467686 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet a765abd702644f079bddffb3fc4238ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a765abd702644f079bddffb3fc4238ef already in progress: creating tablet
I20260924 19:57:36.468403 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet ef3f82de70bb4328a15a044fe0e500bf (table default.Table_cca06980f109487a9ed8b8172950b60d [id=016f4039733144c985808b862eebcc9a]) successfully deleted
I20260924 19:57:36.468979 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 5f994e0401df4fc19bc5abb10f08bc6e (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]) successfully deleted
I20260924 19:57:36.472607 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b7f2dd8ba2a4cb190e9ff92a542bacb" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:36.473385 17816 raft_consensus.cc:3060] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.480962 18224 tablet_bootstrap.cc:492] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:36.482025 18220 tablet_bootstrap.cc:492] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:36.482496 18220 ts_tablet_manager.cc:1403] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.000s
W20260924 19:57:36.471196 17602 consensus_peers.cc:597] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e -> Peer 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309): Couldn't send request to peer 3e059bbe4dac4178bccb3be48cb1118d. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260924 19:57:36.484185 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 07f86f24d9d14d05898e1c0f69428850 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07f86f24d9d14d05898e1c0f69428850 already in progress: creating tablet
I20260924 19:57:36.484758 18224 ts_tablet_manager.cc:1403] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.167s user 0.005s sys 0.005s
I20260924 19:57:36.485064 18220 raft_consensus.cc:359] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.485731 18220 raft_consensus.cc:385] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.486004 18220 raft_consensus.cc:740] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
W20260924 19:57:36.488044 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 114174us
I20260924 19:57:36.497102 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "910d61b8c3574302b5a328187226571e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:36.501843 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6b7f2dd8ba2a4cb190e9ff92a542bacb" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:36.502445 18117 raft_consensus.cc:3060] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.509063 18220 consensus_queue.cc:260] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.515583 18220 ts_tablet_manager.cc:1434] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.033s user 0.004s sys 0.003s
I20260924 19:57:36.516500 18220 tablet_bootstrap.cc:492] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:36.520509 18117 raft_consensus.cc:2468] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:36.523203 18729 ts_tablet_manager.cc:1929] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:36.524966 17905 leader_election.cc:304] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:36.529371 17816 raft_consensus.cc:2468] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:36.523638 18729 log.cc:1199] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/fec9b6ecfb384b528cf71c4cfb2fd4cb
I20260924 19:57:36.533102 18729 ts_tablet_manager.cc:1950] T fec9b6ecfb384b528cf71c4cfb2fd4cb P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:36.535009 18708 raft_consensus.cc:2804] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:36.535580 18708 raft_consensus.cc:697] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
W20260924 19:57:36.537065 17599 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92c0daa7d67e4aa390e334c669adda76 consensus state has no leader
I20260924 19:57:36.536580 18708 consensus_queue.cc:237] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.531471 18636 raft_consensus.cc:515] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.550870 18223 tablet_bootstrap.cc:492] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:36.551347 18223 ts_tablet_manager.cc:1403] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.237s user 0.072s sys 0.117s
I20260924 19:57:36.551421 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet fec9b6ecfb384b528cf71c4cfb2fd4cb (table default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]) successfully deleted
I20260924 19:57:36.551829 18220 tablet_bootstrap.cc:654] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:36.551713 17967 raft_consensus.cc:2393] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eaf64bfb64aa42948c9bc1411ce4ce35 in current term 1: Already voted for candidate f74e6ed351284d7fb110e44336ce643a in this term.
I20260924 19:57:36.558789 18223 raft_consensus.cc:359] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.559468 18223 raft_consensus.cc:385] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.559727 18223 raft_consensus.cc:740] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.567317 18223 consensus_queue.cc:260] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.572662 18223 ts_tablet_manager.cc:1434] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260924 19:57:36.576283 18223 tablet_bootstrap.cc:492] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:36.582072 18223 tablet_bootstrap.cc:654] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:36.593778 18220 tablet_bootstrap.cc:492] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:36.594247 18220 ts_tablet_manager.cc:1403] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.078s user 0.008s sys 0.001s
I20260924 19:57:36.495394 18224 raft_consensus.cc:359] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.595084 18224 raft_consensus.cc:385] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.595424 18224 raft_consensus.cc:740] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.596092 18224 consensus_queue.cc:260] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.596817 18220 raft_consensus.cc:359] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.597412 18220 raft_consensus.cc:385] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.597661 18220 raft_consensus.cc:740] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.598115 18224 ts_tablet_manager.cc:1434] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.113s user 0.003s sys 0.000s
I20260924 19:57:36.598328 18220 consensus_queue.cc:260] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.598953 18224 tablet_bootstrap.cc:492] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:36.558219 18231 ts_tablet_manager.cc:1929] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:36.599313 18231 log.cc:1199] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/ef3f82de70bb4328a15a044fe0e500bf
I20260924 19:57:36.600049 18231 ts_tablet_manager.cc:1950] T ef3f82de70bb4328a15a044fe0e500bf P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:36.614439 17606 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:36.615589 17606 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 1 got 0
I20260924 19:57:36.624321 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:36.624426 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "910d61b8c3574302b5a328187226571e" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:36.625097 17815 raft_consensus.cc:3060] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.625283 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:36.626700 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 1 got 0
I20260924 19:57:36.627748 18726 raft_consensus.cc:493] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:36.628247 18726 raft_consensus.cc:515] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.630327 18726 leader_election.cc:290] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:36.631600 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41bdb846ca19425b882434df0167479e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:36.632285 17821 raft_consensus.cc:2468] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:36.633510 18051 leader_election.cc:304] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:36.634480 18726 raft_consensus.cc:2804] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:36.635072 18726 raft_consensus.cc:493] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:36.635486 18726 raft_consensus.cc:3060] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.635696 18220 ts_tablet_manager.cc:1434] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.041s user 0.006s sys 0.001s
I20260924 19:57:36.638764 17607 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 184) took 1955 ms. Trace:
I20260924 19:57:36.638783 18220 tablet_bootstrap.cc:492] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:36.639091 17607 rpcz_store.cc:276] 0924 19:57:34.683432 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.809918 (+126486us) service_pool.cc:224] Handling call
0924 19:57:34.811052 (+ 1134us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:34.854562 (+ 43510us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:34.872461 (+ 17899us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:34.872708 (+ 247us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:34.915916 (+ 43208us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b5400030300. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa55074e151 00007fa550822d49 00007fa55081896d 00007fa5507d9fcb 00007fa5507d9da4 00007fa55223eaf1 00007fa55223f34f 00007fa5520effe4 00007fa5520ed10a 00007fa5520e51fb 00007fa552207822
0924 19:57:34.970337 (+ 54421us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:34.970367 (+ 30us) write_op.cc:454] Released schema lock
0924 19:57:34.970596 (+ 229us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:36.638680 (+1668084us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:34.918462 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:34.918873 (+ 411us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:34.918910 (+ 37us) write_op.cc:435] Acquired schema lock
0924 19:57:34.918924 (+ 14us) tablet.cc:662] Decoding operations
0924 19:57:34.919236 (+ 312us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:34.919274 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:34.919299 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:34.919447 (+ 148us) tablet.cc:701] Row locks acquired
0924 19:57:34.919461 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:34.919588 (+ 127us) write_op.cc:270] Start()
0924 19:57:34.919671 (+ 83us) write_op.cc:276] Timestamp: P: 1790279854919527 usec, L: 0
0924 19:57:34.919687 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:34.920208 (+ 521us) log.cc:844] Serialized 405 byte log entry
0924 19:57:34.938016 (+ 17808us) op_driver.cc:464] REPLICATION: finished
0924 19:57:34.939309 (+ 1293us) write_op.cc:301] APPLY: starting
0924 19:57:34.939373 (+ 64us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:34.939480 (+ 107us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:34.939593 (+ 113us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:34.939784 (+ 191us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:34.939933 (+ 149us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:34.939980 (+ 47us) write_op.cc:312] APPLY: finished
0924 19:57:34.940169 (+ 189us) log.cc:844] Serialized 23 byte log entry
0924 19:57:34.940311 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:34.940372 (+ 61us) write_op.cc:454] Released schema lock
0924 19:57:34.940404 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":91518336,"child_traces":[["op",{"apply.queue_time_us":1127,"apply.run_cpu_time_us":1288,"apply.run_wall_time_us":1288,"num_ops":1,"prepare.queue_time_us":87,"prepare.run_cpu_time_us":2016,"prepare.run_wall_time_us":2162,"raft.queue_time_us":83,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":18235}]]}
I20260924 19:57:36.644361 17614 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 189) took 1566 ms. Trace:
I20260924 19:57:36.644722 17614 rpcz_store.cc:276] 0924 19:57:35.078375 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:35.909156 (+830781us) service_pool.cc:224] Handling call
0924 19:57:36.059573 (+150417us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:36.065661 (+ 6088us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:36.643732 (+578071us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:36.644026 (+ 294us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:36.644301 (+ 275us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:36.101787 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:36.102188 (+ 401us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:36.102211 (+ 23us) write_op.cc:435] Acquired schema lock
0924 19:57:36.102224 (+ 13us) tablet.cc:662] Decoding operations
0924 19:57:36.102489 (+ 265us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:36.102524 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:36.102544 (+ 20us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:36.102737 (+ 193us) tablet.cc:701] Row locks acquired
0924 19:57:36.102751 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:36.102865 (+ 114us) write_op.cc:270] Start()
0924 19:57:36.102944 (+ 79us) write_op.cc:276] Timestamp: P: 1790279856102786 usec, L: 0
0924 19:57:36.102959 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:57:36.103474 (+ 515us) log.cc:844] Serialized 433 byte log entry
0924 19:57:36.605628 (+502154us) op_driver.cc:464] REPLICATION: finished
0924 19:57:36.622672 (+ 17044us) write_op.cc:301] APPLY: starting
0924 19:57:36.622770 (+ 98us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:36.622938 (+ 168us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:36.623093 (+ 155us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:36.623598 (+ 505us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:36.623743 (+ 145us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:36.623786 (+ 43us) write_op.cc:312] APPLY: finished
0924 19:57:36.623961 (+ 175us) log.cc:844] Serialized 29 byte log entry
0924 19:57:36.643369 (+ 19408us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:36.643482 (+ 113us) write_op.cc:454] Released schema lock
0924 19:57:36.643525 (+ 43us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":16749,"num_ops":2,"prepare.queue_time_us":102,"prepare.run_cpu_time_us":1953,"prepare.run_wall_time_us":1952,"replication_time_us":502518}]]}
I20260924 19:57:36.653502 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41bdb846ca19425b882434df0167479e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:36.656728 18050 leader_election.cc:343] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:36.663920 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 123) took 2632 ms. Trace:
I20260924 19:57:36.664275 18231 rpcz_store.cc:276] 0924 19:57:34.031419 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.044170 (+ 12751us) service_pool.cc:224] Handling call
0924 19:57:36.249066 (+2204896us) ts_tablet_manager.cc:1151] Deleting tablet ef3f82de70bb4328a15a044fe0e500bf
0924 19:57:36.249117 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:36.458085 (+208968us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.558088 (+100003us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.619620 (+ 61532us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b5800000f98. stack: 00007f7c8a13c323 00007f7c8a13c220 00007f7c89ce0574 0000000000516964 00000000005168f9 00007f7c89ff28a1 00007f7c8a136b51 00007f7c8a137182 00007f7c930a8ec9 00007f7c922bb918 00007f7c92382d82 00007f7c923b3960 00007f7c92399e6d 00007f7c92399e19 00007f7c92399da9 00007f7c92399d71
0924 19:57:36.663852 (+ 44232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33711488,"tablet-delete.queue_time_us":2204187}
I20260924 19:57:36.665799 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet ef3f82de70bb4328a15a044fe0e500bf (table default.Table_cca06980f109487a9ed8b8172950b60d [id=016f4039733144c985808b862eebcc9a]) successfully deleted
I20260924 19:57:36.667147 18636 leader_election.cc:290] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:36.547302 18561 raft_consensus.cc:515] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.670457 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "910d61b8c3574302b5a328187226571e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:36.673142 18224 tablet_bootstrap.cc:654] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:36.675959 18561 leader_election.cc:290] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:36.677284 18726 raft_consensus.cc:515] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.674305 17815 raft_consensus.cc:2468] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:36.681270 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41bdb846ca19425b882434df0167479e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:36.678493 18231 tablet_replica.cc:333] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:36.682802 17906 leader_election.cc:304] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:36.683049 18231 raft_consensus.cc:2243] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:36.683677 18231 raft_consensus.cc:2272] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:36.683869 18730 raft_consensus.cc:493] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:36.684124 18636 raft_consensus.cc:2804] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:36.684337 18730 raft_consensus.cc:515] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.686000 18231 ts_tablet_manager.cc:1916] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:36.686441 18730 leader_election.cc:290] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:36.686861 18730 raft_consensus.cc:493] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:36.687672 18730 raft_consensus.cc:515] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.689034 18636 raft_consensus.cc:697] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:36.682097 17816 raft_consensus.cc:3060] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.690058 18730 leader_election.cc:290] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:36.691416 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e68378312c334fc28dafb71bf6386be8" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:36.693235 17821 raft_consensus.cc:2468] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:36.694840 18051 leader_election.cc:304] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:36.697739 18561 raft_consensus.cc:2804] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:36.698863 18561 raft_consensus.cc:493] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:36.700179 18561 raft_consensus.cc:3060] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.698482 18636 consensus_queue.cc:237] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.706314 17816 raft_consensus.cc:2468] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:36.710239 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "910d61b8c3574302b5a328187226571e" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:36.711032 18115 raft_consensus.cc:2393] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:36.712639 18220 tablet_bootstrap.cc:654] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:36.713804 18561 raft_consensus.cc:515] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.714617 18726 leader_election.cc:290] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:36.722079 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "910d61b8c3574302b5a328187226571e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:36.723606 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41bdb846ca19425b882434df0167479e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:36.726312 18223 tablet_bootstrap.cc:492] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:36.726523 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1684aaf9d984649bff8c4b0b12044b9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:36.726728 18223 ts_tablet_manager.cc:1403] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.151s user 0.012s sys 0.006s
I20260924 19:57:36.729010 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e68378312c334fc28dafb71bf6386be8" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:36.729249 18223 raft_consensus.cc:359] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:36.729576 17966 raft_consensus.cc:2468] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:36.729869 18223 raft_consensus.cc:385] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.729954 18051 leader_election.cc:304] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:36.730160 18223 raft_consensus.cc:740] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.707204 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1684aaf9d984649bff8c4b0b12044b9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:36.731990 17815 raft_consensus.cc:2468] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:36.734083 18051 leader_election.cc:304] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:36.740494 18746 raft_consensus.cc:2804] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:36.741824 18726 raft_consensus.cc:2804] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:36.742285 18726 raft_consensus.cc:493] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:36.742671 18726 raft_consensus.cc:3060] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.754115 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e68378312c334fc28dafb71bf6386be8" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:36.754793 17969 raft_consensus.cc:3060] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.762579 17969 raft_consensus.cc:2468] T e68378312c334fc28dafb71bf6386be8 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:36.727782 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e68378312c334fc28dafb71bf6386be8" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:36.728936 18224 tablet_bootstrap.cc:492] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:36.764492 17819 raft_consensus.cc:3060] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.764587 18224 ts_tablet_manager.cc:1403] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.166s user 0.014s sys 0.000s
I20260924 19:57:36.767040 18224 raft_consensus.cc:359] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:36.767689 18224 raft_consensus.cc:385] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.767930 18224 raft_consensus.cc:740] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.768638 18224 consensus_queue.cc:260] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:36.773015 18746 raft_consensus.cc:697] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:36.773247 18753 raft_consensus.cc:493] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:36.773690 18753 raft_consensus.cc:515] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.774103 18746 consensus_queue.cc:237] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.777513 18224 ts_tablet_manager.cc:1434] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20260924 19:57:36.778558 18750 tablet.cc:1751] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260924 19:57:36.780980 18751 tablet.cc:1751] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260924 19:57:36.785470 18561 leader_election.cc:290] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:36.787915 18730 raft_consensus.cc:493] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:36.789156 18730 raft_consensus.cc:515] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.789826 18726 raft_consensus.cc:515] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.780172 18224 tablet_bootstrap.cc:492] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:36.784250 17819 raft_consensus.cc:2468] T e68378312c334fc28dafb71bf6386be8 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:36.797279 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872bc32b9e384c828a0699079b27530d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:36.797945 17966 raft_consensus.cc:2468] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:36.807238 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c861e93ad9f40ba9ad010021e43bc2b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:36.808243 17821 raft_consensus.cc:2468] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:36.809326 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c861e93ad9f40ba9ad010021e43bc2b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:36.795867 18050 leader_election.cc:304] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:36.798547 18730 leader_election.cc:290] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:36.820070 18220 tablet_bootstrap.cc:492] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:36.820519 18220 ts_tablet_manager.cc:1403] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.182s user 0.005s sys 0.005s
I20260924 19:57:36.821798 18224 tablet_bootstrap.cc:654] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:36.816075 18050 leader_election.cc:343] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:36.823457 18701 raft_consensus.cc:493] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:36.824097 18050 leader_election.cc:343] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:36.823864 18701 raft_consensus.cc:515] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:36.825264 18050 leader_election.cc:304] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:36.825769 18701 leader_election.cc:290] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:36.730764 18223 consensus_queue.cc:260] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:36.809288 18051 leader_election.cc:304] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:36.826253 18050 leader_election.cc:304] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:36.827612 18757 raft_consensus.cc:2804] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:36.827729 18636 raft_consensus.cc:493] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:36.828169 18636 raft_consensus.cc:515] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:36.829157 17753 leader_election.cc:343] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: fec9b6ecfb384b528cf71c4cfb2fd4cb
I20260924 19:57:36.815723 18730 raft_consensus.cc:2749] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 19:57:36.830807 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd6e656a653a43d1843a91fee813237d" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:36.831149 18730 raft_consensus.cc:2804] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:36.831492 18730 raft_consensus.cc:493] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:36.831434 17815 raft_consensus.cc:2468] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:36.831782 18730 raft_consensus.cc:3060] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.833055 17906 leader_election.cc:304] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:36.833786 18708 raft_consensus.cc:2804] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:36.834138 18708 raft_consensus.cc:493] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:36.834474 18708 raft_consensus.cc:3060] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.837085 18231 ts_tablet_manager.cc:1929] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:36.837519 18231 log.cc:1199] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/5f994e0401df4fc19bc5abb10f08bc6e
I20260924 19:57:36.838253 18231 ts_tablet_manager.cc:1950] T 5f994e0401df4fc19bc5abb10f08bc6e P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:36.839460 18752 raft_consensus.cc:2804] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:36.839830 18752 raft_consensus.cc:493] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:36.840157 18752 raft_consensus.cc:3060] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.840500 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1684aaf9d984649bff8c4b0b12044b9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:36.841046 17821 raft_consensus.cc:3060] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.817497 18726 leader_election.cc:290] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:36.812727 18753 leader_election.cc:290] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:36.845062 18050 leader_election.cc:343] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:36.845342 18223 ts_tablet_manager.cc:1434] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.118s user 0.007s sys 0.000s
I20260924 19:57:36.846815 18223 tablet_bootstrap.cc:492] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:36.849138 18636 leader_election.cc:290] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:36.852196 18708 raft_consensus.cc:515] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.855108 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd6e656a653a43d1843a91fee813237d" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:36.855657 17815 raft_consensus.cc:3060] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.856865 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 125) took 2269 ms. Trace:
I20260924 19:57:36.859874 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd6e656a653a43d1843a91fee813237d" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:36.862831 18223 tablet_bootstrap.cc:654] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:36.862696 17905 leader_election.cc:343] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:36.864898 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1684aaf9d984649bff8c4b0b12044b9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:36.866155 18231 rpcz_store.cc:276] 0924 19:57:34.586959 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.587078 (+ 119us) service_pool.cc:224] Handling call
0924 19:57:36.678296 (+2091218us) ts_tablet_manager.cc:1151] Deleting tablet 5f994e0401df4fc19bc5abb10f08bc6e
0924 19:57:36.678362 (+ 66us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:36.750475 (+ 72113us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.836943 (+ 86468us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:36.856787 (+ 19844us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2090519}
I20260924 19:57:36.867551 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 5f994e0401df4fc19bc5abb10f08bc6e (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]) successfully deleted
I20260924 19:57:36.868126 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 131) took 1283 ms. Trace:
I20260924 19:57:36.868445 18231 rpcz_store.cc:276] 0924 19:57:35.585126 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:35.616240 (+ 31114us) service_pool.cc:224] Handling call
0924 19:57:36.867744 (+1251504us) ts_tablet_manager.cc:1151] Deleting tablet 07f86f24d9d14d05898e1c0f69428850
0924 19:57:36.867792 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:36.868062 (+ 270us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1250425}
W20260924 19:57:36.869547 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 07f86f24d9d14d05898e1c0f69428850 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07f86f24d9d14d05898e1c0f69428850 already in progress: creating tablet
I20260924 19:57:36.872583 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fec9b6ecfb384b528cf71c4cfb2fd4cb" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:36.875172 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872bc32b9e384c828a0699079b27530d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:36.875315 18220 raft_consensus.cc:359] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.876003 18220 raft_consensus.cc:385] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.876303 18220 raft_consensus.cc:740] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
W20260924 19:57:36.876904 17753 leader_election.cc:343] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:36.877049 18220 consensus_queue.cc:260] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.877460 17753 leader_election.cc:304] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:36.878446 18701 raft_consensus.cc:2749] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:36.877645 17819 raft_consensus.cc:2468] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
W20260924 19:57:36.880347 18050 leader_election.cc:343] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:36.879069 18231 tablet_replica.cc:333] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:36.887450 17821 raft_consensus.cc:2468] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:36.888942 18708 leader_election.cc:290] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:36.887584 18231 raft_consensus.cc:2243] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:36.889762 18231 raft_consensus.cc:2272] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:36.891103 18752 raft_consensus.cc:515] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.891548 18051 leader_election.cc:304] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:36.892419 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd6e656a653a43d1843a91fee813237d" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:36.892709 18746 raft_consensus.cc:2804] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:36.893338 18746 raft_consensus.cc:697] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
W20260924 19:57:36.894225 17905 leader_election.cc:343] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:36.887398 17815 raft_consensus.cc:2468] T fd6e656a653a43d1843a91fee813237d P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:36.896423 17906 leader_election.cc:304] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters: eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:36.894268 18746 consensus_queue.cc:237] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.897920 18761 tablet.cc:1751] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20260924 19:57:36.902700 18730 raft_consensus.cc:515] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.905867 18730 leader_election.cc:290] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:36.906576 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c861e93ad9f40ba9ad010021e43bc2b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:36.907357 17815 raft_consensus.cc:3060] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.878975 18220 ts_tablet_manager.cc:1434] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.058s user 0.003s sys 0.003s
I20260924 19:57:36.899473 18708 raft_consensus.cc:2804] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:36.907931 18757 raft_consensus.cc:697] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:36.908895 18220 tablet_bootstrap.cc:492] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:36.909073 18757 consensus_queue.cc:237] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.911669 18231 ts_tablet_manager.cc:1916] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:36.901530 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872bc32b9e384c828a0699079b27530d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:36.914683 17966 raft_consensus.cc:3060] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.905885 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c861e93ad9f40ba9ad010021e43bc2b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:36.923444 18708 raft_consensus.cc:697] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:36.924345 18708 consensus_queue.cc:237] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.925395 18223 tablet_bootstrap.cc:492] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:36.925899 18223 ts_tablet_manager.cc:1403] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.079s user 0.008s sys 0.002s
I20260924 19:57:36.928858 18223 raft_consensus.cc:359] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:36.929528 18223 raft_consensus.cc:385] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.929795 18223 raft_consensus.cc:740] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.930437 18223 consensus_queue.cc:260] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:36.934139 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872bc32b9e384c828a0699079b27530d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:36.934795 17821 raft_consensus.cc:3060] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:36.939254 18752 leader_election.cc:290] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:36.946475 18050 leader_election.cc:343] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:36.949470 17815 raft_consensus.cc:2468] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:36.950666 18051 leader_election.cc:304] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:36.954543 17966 raft_consensus.cc:2468] T 872bc32b9e384c828a0699079b27530d P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:36.955528 18757 raft_consensus.cc:2804] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:36.958523 18220 tablet_bootstrap.cc:654] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:36.959594 18757 raft_consensus.cc:697] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:36.960970 18757 consensus_queue.cc:237] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.965788 18050 leader_election.cc:304] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:36.966500 18746 raft_consensus.cc:2804] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:36.966982 18746 raft_consensus.cc:697] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:36.967803 18746 consensus_queue.cc:237] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:36.969870 17821 raft_consensus.cc:2468] T 872bc32b9e384c828a0699079b27530d P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:36.972626 18224 tablet_bootstrap.cc:492] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:36.973263 18224 ts_tablet_manager.cc:1403] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.194s user 0.006s sys 0.003s
I20260924 19:57:36.977391 18223 ts_tablet_manager.cc:1434] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.051s user 0.006s sys 0.000s
I20260924 19:57:36.979118 18223 tablet_bootstrap.cc:492] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:36.977003 18224 raft_consensus.cc:359] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:36.982790 18224 raft_consensus.cc:385] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:36.983072 18224 raft_consensus.cc:740] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:36.984033 18224 consensus_queue.cc:260] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.033674 17946 tablet_service.cc:1511] Processing CreateTablet for tablet 8aef0359a77240679240823612507ee8 (DEFAULT_TABLE table=default.Table_5aeda90fd8e3425a8c49a239533690c9 [id=63622937f5a64a05b56aa1310407ec99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.034919 17946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aef0359a77240679240823612507ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.035509 17607 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:37.036635 17607 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 1 got 0
I20260924 19:57:37.036777 18636 raft_consensus.cc:493] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:37.037415 17607 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]). Expected version 5 got 4
I20260924 19:57:37.037204 18636 raft_consensus.cc:515] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.037752 18231 ts_tablet_manager.cc:1929] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:37.038117 18231 log.cc:1199] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/fec9b6ecfb384b528cf71c4cfb2fd4cb
I20260924 19:57:37.038453 18220 tablet_bootstrap.cc:492] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:37.038363 17607 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:37.038727 18231 ts_tablet_manager.cc:1950] T fec9b6ecfb384b528cf71c4cfb2fd4cb P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:37.038846 18220 ts_tablet_manager.cc:1403] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.130s user 0.006s sys 0.002s
I20260924 19:57:37.041188 18095 tablet_service.cc:1511] Processing CreateTablet for tablet 8aef0359a77240679240823612507ee8 (DEFAULT_TABLE table=default.Table_5aeda90fd8e3425a8c49a239533690c9 [id=63622937f5a64a05b56aa1310407ec99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.041656 17796 tablet_service.cc:1511] Processing CreateTablet for tablet 8aef0359a77240679240823612507ee8 (DEFAULT_TABLE table=default.Table_5aeda90fd8e3425a8c49a239533690c9 [id=63622937f5a64a05b56aa1310407ec99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.042225 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aef0359a77240679240823612507ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.042789 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aef0359a77240679240823612507ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.044009 18224 ts_tablet_manager.cc:1434] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.070s user 0.009s sys 0.000s
I20260924 19:57:37.044965 18224 tablet_bootstrap.cc:492] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:37.044976 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 924bc897f48b44069f1fb2ee71e26db7 (DEFAULT_TABLE table=default.Table_415dca00ed274e59961ddf45e42de178 [id=b8a8b30617164fbcb9f6918142768362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.048017 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41bdb846ca19425b882434df0167479e" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:37.048732 17815 raft_consensus.cc:2393] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:37.051504 17795 tablet_service.cc:1511] Processing CreateTablet for tablet 924bc897f48b44069f1fb2ee71e26db7 (DEFAULT_TABLE table=default.Table_415dca00ed274e59961ddf45e42de178 [id=b8a8b30617164fbcb9f6918142768362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.052563 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 924bc897f48b44069f1fb2ee71e26db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.054242 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]). Expected version 5 got 4
I20260924 19:57:37.054955 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:37.055608 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:37.056440 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:37.058702 17797 tablet_service.cc:1511] Processing CreateTablet for tablet 3192f66a492442c1bf4137cfe7e13bc9 (DEFAULT_TABLE table=default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b [id=7babe5418e104e76871835ca3e30f95c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.059309 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 3192f66a492442c1bf4137cfe7e13bc9 (DEFAULT_TABLE table=default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b [id=7babe5418e104e76871835ca3e30f95c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.059979 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:37.061117 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet c1684aaf9d984649bff8c4b0b12044b9 (table default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe]). Expected version 1 got 0
I20260924 19:57:37.061991 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_b2d71701938e43c2bb30b8594cb20cf0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 1 got 0
I20260924 19:57:37.062160 18224 tablet_bootstrap.cc:654] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:37.039139 18636 leader_election.cc:290] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:37.066006 18098 tablet_service.cc:1511] Processing CreateTablet for tablet ae7a0660795e4e4b87d0959ca84d5d1b (DEFAULT_TABLE table=default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.041285 18220 raft_consensus.cc:359] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.066783 18220 raft_consensus.cc:385] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:37.050733 17942 tablet_service.cc:1511] Processing CreateTablet for tablet 924bc897f48b44069f1fb2ee71e26db7 (DEFAULT_TABLE table=default.Table_415dca00ed274e59961ddf45e42de178 [id=b8a8b30617164fbcb9f6918142768362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.068900 17942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 924bc897f48b44069f1fb2ee71e26db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.069049 17799 tablet_service.cc:1511] Processing CreateTablet for tablet ae7a0660795e4e4b87d0959ca84d5d1b (DEFAULT_TABLE table=default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.069798 17611 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_5b3a0213ccbb4622a13a68e24d12af4e [id=54f05b78c3324978b2635be4bd969f66] alter complete (version 1)
I20260924 19:57:37.063362 17948 tablet_service.cc:1511] Processing CreateTablet for tablet ae7a0660795e4e4b87d0959ca84d5d1b (DEFAULT_TABLE table=default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.071830 18223 tablet_bootstrap.cc:654] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:37.072278 17948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7a0660795e4e4b87d0959ca84d5d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.073443 17612 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 182) took 2606 ms. Trace:
I20260924 19:57:37.067052 18220 raft_consensus.cc:740] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:37.075122 18220 consensus_queue.cc:260] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.076663 17801 tablet_service.cc:1511] Processing CreateTablet for tablet f43f2a255c644ca6a68a8d867798fc1a (DEFAULT_TABLE table=default.Table_fe4242bbbc1f4747bd1f2b4d25c8bacd [id=fa0b10df171340868b0305a28d192ea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.077629 18220 ts_tablet_manager.cc:1434] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.039s user 0.005s sys 0.003s
I20260924 19:57:37.062466 17943 tablet_service.cc:1511] Processing CreateTablet for tablet 3192f66a492442c1bf4137cfe7e13bc9 (DEFAULT_TABLE table=default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b [id=7babe5418e104e76871835ca3e30f95c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.078625 18089 tablet_service.cc:1511] Processing CreateTablet for tablet f43f2a255c644ca6a68a8d867798fc1a (DEFAULT_TABLE table=default.Table_fe4242bbbc1f4747bd1f2b4d25c8bacd [id=fa0b10df171340868b0305a28d192ea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.073762 17612 rpcz_store.cc:276] 0924 19:57:34.467508 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.475685 (+ 8177us) service_pool.cc:224] Handling call
0924 19:57:34.517618 (+ 41933us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:34.518656 (+ 1038us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:34.527060 (+ 8404us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:34.527298 (+ 238us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:34.847331 (+320033us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:37.073357 (+2226026us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:34.539311 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:34.539791 (+ 480us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:34.539823 (+ 32us) write_op.cc:435] Acquired schema lock
0924 19:57:34.539843 (+ 20us) tablet.cc:662] Decoding operations
0924 19:57:34.540119 (+ 276us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:34.540155 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:34.540181 (+ 26us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:34.540318 (+ 137us) tablet.cc:701] Row locks acquired
0924 19:57:34.540330 (+ 12us) write_op.cc:260] PREPARE: finished
0924 19:57:34.540438 (+ 108us) write_op.cc:270] Start()
0924 19:57:34.540525 (+ 87us) write_op.cc:276] Timestamp: P: 1790279854540368 usec, L: 0
0924 19:57:34.540540 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:57:34.559723 (+ 19183us) log.cc:844] Serialized 405 byte log entry
0924 19:57:34.561095 (+ 1372us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f7aa1 00007fa55038a5e8 00007fa55081a741 00007fa550819f18 00007fa55081ebb1 00007fa55081eb69 00007fa55081eaf9 00007fa55081eac1 00007fa55081dced 00000000005bd4e4 00000000005bd319
0924 19:57:34.844628 (+283533us) op_driver.cc:464] REPLICATION: finished
0924 19:57:34.845577 (+ 949us) write_op.cc:301] APPLY: starting
0924 19:57:34.845638 (+ 61us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:34.845735 (+ 97us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:34.845761 (+ 26us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:34.845998 (+ 237us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:34.846127 (+ 129us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:34.846171 (+ 44us) write_op.cc:312] APPLY: finished
0924 19:57:34.846352 (+ 181us) log.cc:844] Serialized 23 byte log entry
0924 19:57:34.846497 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:34.846559 (+ 62us) write_op.cc:454] Released schema lock
0924 19:57:34.846591 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:34.846848 (+ 257us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:34.846880 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":792,"apply.run_cpu_time_us":1216,"apply.run_wall_time_us":1216,"mutex_wait_us":8081,"num_ops":1,"prepare.queue_time_us":8237,"prepare.run_cpu_time_us":3559,"prepare.run_wall_time_us":21919,"replication_time_us":303997,"spinlock_wait_cycles":42822528}]]}
I20260924 19:57:37.082687 18092 tablet_service.cc:1511] Processing CreateTablet for tablet 0b99ff1920274831b58d8283bd592818 (DEFAULT_TABLE table=default.Table_aeac9df2b26b490a83f10d672335d75e [id=80222cdc26da492e9f386e9eb2fb77b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.083751 18220 tablet_bootstrap.cc:492] T c1684aaf9d984649bff8c4b0b12044b9 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:37.079003 17943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3192f66a492442c1bf4137cfe7e13bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.088861 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 0b99ff1920274831b58d8283bd592818 (DEFAULT_TABLE table=default.Table_aeac9df2b26b490a83f10d672335d75e [id=80222cdc26da492e9f386e9eb2fb77b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.089406 17947 tablet_service.cc:1511] Processing CreateTablet for tablet f43f2a255c644ca6a68a8d867798fc1a (DEFAULT_TABLE table=default.Table_fe4242bbbc1f4747bd1f2b4d25c8bacd [id=fa0b10df171340868b0305a28d192ea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.090360 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43f2a255c644ca6a68a8d867798fc1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.092324 17950 tablet_service.cc:1511] Processing CreateTablet for tablet 0b99ff1920274831b58d8283bd592818 (DEFAULT_TABLE table=default.Table_aeac9df2b26b490a83f10d672335d75e [id=80222cdc26da492e9f386e9eb2fb77b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.093233 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b99ff1920274831b58d8283bd592818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.095788 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43f2a255c644ca6a68a8d867798fc1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.099466 18092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b99ff1920274831b58d8283bd592818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.100805 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7a0660795e4e4b87d0959ca84d5d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.101974 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 924bc897f48b44069f1fb2ee71e26db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.103338 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3192f66a492442c1bf4137cfe7e13bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.109490 17949 tablet_service.cc:1511] Processing CreateTablet for tablet 484517ee7df8463bb70d201ead1c6835 (DEFAULT_TABLE table=default.Table_771765c449ce492986addb028e1b11d1 [id=f954ad85289641fda971e5c5155b9147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.110558 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 484517ee7df8463bb70d201ead1c6835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.120229 17797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3192f66a492442c1bf4137cfe7e13bc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.126976 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7a0660795e4e4b87d0959ca84d5d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.127909 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 484517ee7df8463bb70d201ead1c6835 (DEFAULT_TABLE table=default.Table_771765c449ce492986addb028e1b11d1 [id=f954ad85289641fda971e5c5155b9147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.128964 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 484517ee7df8463bb70d201ead1c6835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.145376 17612 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]). Expected version 5 got 4
I20260924 19:57:37.145838 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41bdb846ca19425b882434df0167479e" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:37.148164 18708 consensus_queue.cc:1048] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:37.149506 17949 tablet_service.cc:1511] Processing CreateTablet for tablet 431d298f513d47268284d50af2ea1436 (DEFAULT_TABLE table=default.Table_6b68af48dd4140dda7d1854a3cf2269f [id=1a5ed525f4364782a0793571d4caa5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.150779 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431d298f513d47268284d50af2ea1436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.156308 17802 tablet_service.cc:1511] Processing CreateTablet for tablet 484517ee7df8463bb70d201ead1c6835 (DEFAULT_TABLE table=default.Table_771765c449ce492986addb028e1b11d1 [id=f954ad85289641fda971e5c5155b9147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.160019 17905 leader_election.cc:304] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:37.161036 18223 tablet_bootstrap.cc:492] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:37.161087 18708 raft_consensus.cc:3060] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:37.161494 18223 ts_tablet_manager.cc:1403] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.183s user 0.004s sys 0.009s
I20260924 19:57:37.162945 18090 tablet_service.cc:1511] Processing CreateTablet for tablet 431d298f513d47268284d50af2ea1436 (DEFAULT_TABLE table=default.Table_6b68af48dd4140dda7d1854a3cf2269f [id=1a5ed525f4364782a0793571d4caa5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.164062 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431d298f513d47268284d50af2ea1436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.165082 18224 tablet_bootstrap.cc:492] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:37.165473 18224 ts_tablet_manager.cc:1403] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.121s user 0.010s sys 0.005s
I20260924 19:57:37.146914 17612 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:37.167439 17612 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_22092e7254a9428e81176f410a135394 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 2 got 0
I20260924 19:57:37.167784 18224 raft_consensus.cc:359] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.168390 18224 raft_consensus.cc:385] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:37.168642 18224 raft_consensus.cc:740] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:37.169555 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43f2a255c644ca6a68a8d867798fc1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.174422 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b99ff1920274831b58d8283bd592818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.175916 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 135) took 1274 ms. Trace:
I20260924 19:57:37.176267 18231 rpcz_store.cc:276] 0924 19:57:35.901804 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:35.902066 (+ 262us) service_pool.cc:224] Handling call
0924 19:57:36.878914 (+976848us) ts_tablet_manager.cc:1151] Deleting tablet fec9b6ecfb384b528cf71c4cfb2fd4cb
0924 19:57:36.878959 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:36.991737 (+112778us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:37.037627 (+ 45890us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:37.175851 (+138224us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":975816}
I20260924 19:57:37.178135 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 484517ee7df8463bb70d201ead1c6835. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.190028 17612 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:37.190716 17612 catalog_manager.cc:5838] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:37.195843 17947 tablet_service.cc:1511] Processing CreateTablet for tablet ca37ddd22f5d4bcab2862553408447ce (DEFAULT_TABLE table=default.Table_ca51e47e4fc44d9ba5dd07cbeefa6e68 [id=6ea0f9084fd84e848fdecdd696cdc773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.197038 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca37ddd22f5d4bcab2862553408447ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.203948 17613 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 349c6e9413bb41268d0c41d549203e2a
I20260924 19:57:37.205483 17613 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 165) took 4011 ms. Trace:
I20260924 19:57:37.209133 18223 raft_consensus.cc:359] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.209770 18223 raft_consensus.cc:385] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:37.210009 18223 raft_consensus.cc:740] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:37.210037 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet fec9b6ecfb384b528cf71c4cfb2fd4cb (table default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]) successfully deleted
I20260924 19:57:37.211725 17796 tablet_service.cc:1511] Processing CreateTablet for tablet 431d298f513d47268284d50af2ea1436 (DEFAULT_TABLE table=default.Table_6b68af48dd4140dda7d1854a3cf2269f [id=1a5ed525f4364782a0793571d4caa5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.212812 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431d298f513d47268284d50af2ea1436. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.217293 17944 tablet_service.cc:1558] Processing DeleteTablet for tablet 349c6e9413bb41268d0c41d549203e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:37.220592 17599 catalog_manager.cc:5136] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): alter failed for tablet 5f994e0401df4fc19bc5abb10f08bc6e (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]),no further retry: Not found: Tablet not found: 5f994e0401df4fc19bc5abb10f08bc6e
I20260924 19:57:37.221565 18701 raft_consensus.cc:493] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:37.222014 18701 raft_consensus.cc:515] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.224015 18701 leader_election.cc:290] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:37.227259 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b328d37d55e4116a29322caaa21d431" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:37.227339 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b328d37d55e4116a29322caaa21d431" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:37.227936 18115 raft_consensus.cc:2468] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
W20260924 19:57:37.228797 17753 leader_election.cc:343] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.229487 17753 leader_election.cc:304] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:37.218920 17606 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_850e5158d78d4850b87445dac75ef7a9 [id=f768ce157a7b407d9f1e55a5b9d0720a] alter complete (version 1)
I20260924 19:57:37.233665 18087 tablet_service.cc:1558] Processing DeleteTablet for tablet 349c6e9413bb41268d0c41d549203e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:37.256204 17952 tablet_service.cc:1558] Processing DeleteTablet for tablet efd7339d0fee436299a7eac02a82addd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:37.256824 17951 tablet_service.cc:1558] Processing DeleteTablet for tablet a2e4519aed7642678db3a518a89778de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:37.205777 17613 rpcz_store.cc:276] 0924 19:57:33.193957 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.194406 (+ 449us) service_pool.cc:224] Handling call
0924 19:57:33.355461 (+161055us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:33.355977 (+ 516us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:33.356145 (+ 168us) catalog_manager.cc:2863] Locking tablets
0924 19:57:34.248686 (+892541us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:34.550988 (+302302us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:34.551246 (+ 258us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:37.203853 (+2652607us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:37.205422 (+ 1569us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:34.373329 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:34.373677 (+ 348us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:34.373700 (+ 23us) write_op.cc:435] Acquired schema lock
0924 19:57:34.373712 (+ 12us) tablet.cc:662] Decoding operations
0924 19:57:34.374021 (+ 309us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:34.374061 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:34.374086 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:34.374358 (+ 272us) tablet.cc:701] Row locks acquired
0924 19:57:34.374374 (+ 16us) write_op.cc:260] PREPARE: finished
0924 19:57:34.374476 (+ 102us) write_op.cc:270] Start()
0924 19:57:34.374556 (+ 80us) write_op.cc:276] Timestamp: P: 1790279854374424 usec, L: 0
0924 19:57:34.374573 (+ 17us) op_driver.cc:348] REPLICATION: starting
0924 19:57:34.375015 (+ 442us) log.cc:844] Serialized 793 byte log entry
0924 19:57:34.538695 (+163680us) op_driver.cc:464] REPLICATION: finished
0924 19:57:34.541768 (+ 3073us) write_op.cc:301] APPLY: starting
0924 19:57:34.541815 (+ 47us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:34.541934 (+ 119us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:34.542052 (+ 118us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:34.542307 (+ 255us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:34.542449 (+ 142us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:34.542500 (+ 51us) write_op.cc:312] APPLY: finished
0924 19:57:34.542685 (+ 185us) log.cc:844] Serialized 29 byte log entry
0924 19:57:34.542825 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:34.542911 (+ 86us) write_op.cc:454] Released schema lock
0924 19:57:34.542943 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:34.551448 (+ 8505us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:34.551478 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2941,"apply.run_cpu_time_us":1396,"apply.run_wall_time_us":9636,"mutex_wait_us":118270,"num_ops":2,"prepare.queue_time_us":122754,"prepare.run_cpu_time_us":1937,"prepare.run_wall_time_us":3084,"raft.queue_time_us":80,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":164047}]]}
I20260924 19:57:37.257436 17941 tablet_service.cc:1558] Processing DeleteTablet for tablet ef3f82de70bb4328a15a044fe0e500bf with delete_type TABLET_DATA_DELETED (replaced by tablet 57008266b80345e0a3f8e60bd7e2d62b) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:37.258018 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c2440e1cade4327a63d6234e3adcd64 with delete_type TABLET_DATA_DELETED (replaced by tablet 07f86f24d9d14d05898e1c0f69428850) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:37.268505 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 151247d6b5254cd884c58db9311ac04a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:37.271688 17609 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 33) took 4065 ms (client timeout 1000 ms). Trace:
I20260924 19:57:37.271976 17941 tablet_service.cc:1558] Processing DeleteTablet for tablet 085b8088134d47478f0ce06a6a9f8572 with delete_type TABLET_DATA_DELETED (replaced by tablet fec9b6ecfb384b528cf71c4cfb2fd4cb) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:37.272976 17941 tablet_service.cc:1558] Processing DeleteTablet for tablet f482b53a4a2d44f6b3431c588cd605ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:37.274137 17945 tablet_service.cc:1558] Processing DeleteTablet for tablet 916877c7540a4f6685064fc2807cb140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:37.274730 18701 raft_consensus.cc:2804] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:37.275168 18701 raft_consensus.cc:493] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:37.275611 18701 raft_consensus.cc:3060] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:37.277128 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet 349c6e9413bb41268d0c41d549203e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.278244 18231 tablet_replica.cc:333] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:37.279013 18231 raft_consensus.cc:2243] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:37.279440 18223 consensus_queue.cc:260] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.281414 18223 ts_tablet_manager.cc:1434] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.120s user 0.004s sys 0.003s
I20260924 19:57:37.282243 18223 tablet_bootstrap.cc:492] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:37.283563 17600 catalog_manager.cc:5142] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): alter failed for tablet 349c6e9413bb41268d0c41d549203e2a (table default.Table_41c56bb86db34f16aba6e94147cde1d2 [id=b1d9f8b4039b4d19bfc493355e0d07cc]): Illegal state: Tablet not RUNNING: STOPPING
I20260924 19:57:37.245222 18224 consensus_queue.cc:260] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.291661 18231 raft_consensus.cc:2272] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:37.293708 18231 ts_tablet_manager.cc:1916] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:37.272255 17609 rpcz_store.cc:269] 0924 19:57:33.206053 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.206171 (+ 118us) service_pool.cc:224] Handling call
0924 19:57:37.271598 (+4065427us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:34.074640 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:34.075140 (+ 500us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:34.075195 (+ 55us) write_op.cc:435] Acquired schema lock
0924 19:57:34.075214 (+ 19us) tablet.cc:662] Decoding operations
0924 19:57:34.075533 (+ 319us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:34.075585 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:34.075617 (+ 32us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:34.075800 (+ 183us) tablet.cc:701] Row locks acquired
0924 19:57:34.075819 (+ 19us) write_op.cc:260] PREPARE: finished
0924 19:57:34.075953 (+ 134us) write_op.cc:270] Start()
0924 19:57:34.076088 (+ 135us) write_op.cc:276] Timestamp: P: 1790279854071580 usec, L: 0
0924 19:57:34.076110 (+ 22us) op_driver.cc:348] REPLICATION: starting
0924 19:57:34.076701 (+ 591us) log.cc:844] Serialized 530 byte log entry
0924 19:57:34.187766 (+111065us) op_driver.cc:464] REPLICATION: finished
0924 19:57:34.190265 (+ 2499us) write_op.cc:301] APPLY: starting
0924 19:57:34.190323 (+ 58us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:34.190418 (+ 95us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:34.190510 (+ 92us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:34.190679 (+ 169us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:34.190803 (+ 124us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:34.190845 (+ 42us) write_op.cc:312] APPLY: finished
0924 19:57:34.191029 (+ 184us) log.cc:844] Serialized 23 byte log entry
0924 19:57:34.191200 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:34.191261 (+ 61us) write_op.cc:454] Released schema lock
0924 19:57:34.191293 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:34.215974 (+ 24681us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:34.216002 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":47987,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":2331,"apply.run_cpu_time_us":1264,"apply.run_wall_time_us":25664,"num_ops":1,"prepare.queue_time_us":3196,"prepare.run_cpu_time_us":2393,"prepare.run_wall_time_us":2392,"raft.queue_time_us":85,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":15,"replication_time_us":111550,"spinlock_wait_cycles":352128}]]}
I20260924 19:57:37.313391 18224 ts_tablet_manager.cc:1434] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.148s user 0.006s sys 0.001s
I20260924 19:57:37.315868 18224 tablet_bootstrap.cc:492] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:37.339227 17608 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_56ddb56503844349ba42c68ebb67a694 [id=522c4df3169f4d2ba1cbe15df69d40b3] alter complete (version 1)
W20260924 19:57:37.343987 17608 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 35) took 1851 ms (client timeout 1000 ms). Trace:
I20260924 19:57:37.348802 18778 consensus_queue.cc:1048] T 0cf2df461598411f8ad9f3a51eafc6f7 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:37.317292 18782 raft_consensus.cc:493] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:37.350256 18220 tablet_bootstrap.cc:654] T c1684aaf9d984649bff8c4b0b12044b9 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:37.350145 18782 raft_consensus.cc:515] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.353435 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "077f70bcee7e412299da12ddc30a63ff" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:37.355296 17753 leader_election.cc:343] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.363749 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "077f70bcee7e412299da12ddc30a63ff" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:37.365998 17753 leader_election.cc:343] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.366638 17753 leader_election.cc:304] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:37.367599 18780 raft_consensus.cc:2749] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:37.371253 18782 leader_election.cc:290] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:37.377138 17802 tablet_service.cc:1558] Processing DeleteTablet for tablet efd7339d0fee436299a7eac02a82addd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.378319 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet 916877c7540a4f6685064fc2807cb140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.381210 18224 tablet_bootstrap.cc:654] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:37.385494 18751 tablet.cc:1751] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:37.397769 18708 raft_consensus.cc:2749] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:37.399061 18223 tablet_bootstrap.cc:654] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:37.350131 17606 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 25) took 2173 ms (client timeout 1000 ms). Trace:
I20260924 19:57:37.407553 18761 tablet.cc:1751] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260924 19:57:37.407426 17606 rpcz_store.cc:269] 0924 19:57:35.177111 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:35.296560 (+119449us) service_pool.cc:224] Handling call
0924 19:57:37.350054 (+2053494us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:36.752820 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:36.753298 (+ 478us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:36.753324 (+ 26us) write_op.cc:435] Acquired schema lock
0924 19:57:36.753340 (+ 16us) tablet.cc:662] Decoding operations
0924 19:57:36.753636 (+ 296us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:36.753685 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:36.753715 (+ 30us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:36.753884 (+ 169us) tablet.cc:701] Row locks acquired
0924 19:57:36.753901 (+ 17us) write_op.cc:260] PREPARE: finished
0924 19:57:36.754069 (+ 168us) write_op.cc:270] Start()
0924 19:57:36.754175 (+ 106us) write_op.cc:276] Timestamp: P: 1790279856753968 usec, L: 0
0924 19:57:36.754195 (+ 20us) op_driver.cc:348] REPLICATION: starting
0924 19:57:36.754775 (+ 580us) log.cc:844] Serialized 356 byte log entry
0924 19:57:36.779313 (+ 24538us) op_driver.cc:464] REPLICATION: finished
0924 19:57:36.779772 (+ 459us) write_op.cc:301] APPLY: starting
0924 19:57:36.779831 (+ 59us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:36.779920 (+ 89us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:36.780035 (+ 115us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:36.780201 (+ 166us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:36.780334 (+ 133us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:36.780376 (+ 42us) write_op.cc:312] APPLY: finished
0924 19:57:36.780561 (+ 185us) log.cc:844] Serialized 23 byte log entry
0924 19:57:36.780699 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:36.780761 (+ 62us) write_op.cc:454] Released schema lock
0924 19:57:36.780790 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:36.781015 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:36.781040 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":13271,"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":308,"apply.run_cpu_time_us":1197,"apply.run_wall_time_us":1198,"num_ops":1,"prepare.queue_time_us":117422,"prepare.run_cpu_time_us":2346,"prepare.run_wall_time_us":22769,"raft.queue_time_us":15328,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":15,"replication_time_us":25016,"thread_start_us":116857,"threads_started":1}]]}
I20260924 19:57:37.410786 17606 master_service.cc:960] ReplaceTablet: received request to replace tablet 07f86f24d9d14d05898e1c0f69428850 from {username='slave'} at 127.0.0.1:51280
W20260924 19:57:37.351961 17610 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 34) took 2896 ms (client timeout 1000 ms). Trace:
I20260924 19:57:37.411579 18746 raft_consensus.cc:493] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:37.412029 18746 raft_consensus.cc:515] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.412971 18224 tablet_bootstrap.cc:492] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:37.413378 18224 ts_tablet_manager.cc:1403] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.098s user 0.007s sys 0.003s
I20260924 19:57:37.379113 18776 tablet_replica.cc:333] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:37.414115 18746 leader_election.cc:290] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:37.414255 18776 raft_consensus.cc:2243] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 19:57:37.411512 17610 rpcz_store.cc:269] 0924 19:57:34.455806 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.455951 (+ 145us) service_pool.cc:224] Handling call
0924 19:57:37.351894 (+2895943us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:35.917495 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:35.918142 (+ 647us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:35.918186 (+ 44us) write_op.cc:435] Acquired schema lock
0924 19:57:35.918210 (+ 24us) tablet.cc:662] Decoding operations
0924 19:57:35.918790 (+ 580us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:35.918854 (+ 64us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:35.918927 (+ 73us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:35.919438 (+ 511us) tablet.cc:701] Row locks acquired
0924 19:57:35.919463 (+ 25us) write_op.cc:260] PREPARE: finished
0924 19:57:35.919643 (+ 180us) write_op.cc:270] Start()
0924 19:57:35.919780 (+ 137us) write_op.cc:276] Timestamp: P: 1790279855919564 usec, L: 0
0924 19:57:35.919807 (+ 27us) op_driver.cc:348] REPLICATION: starting
0924 19:57:35.920631 (+ 824us) log.cc:844] Serialized 907 byte log entry
0924 19:57:36.588309 (+667678us) op_driver.cc:464] REPLICATION: finished
0924 19:57:36.601273 (+ 12964us) write_op.cc:301] APPLY: starting
0924 19:57:36.601455 (+ 182us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:36.602142 (+ 687us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:36.602269 (+ 127us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:36.602582 (+ 313us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:36.602739 (+ 157us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:36.602790 (+ 51us) write_op.cc:312] APPLY: finished
0924 19:57:36.603014 (+ 224us) log.cc:844] Serialized 29 byte log entry
0924 19:57:36.604564 (+ 1550us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:36.604660 (+ 96us) write_op.cc:454] Released schema lock
0924 19:57:36.604708 (+ 48us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":12760,"apply.run_cpu_time_us":1798,"apply.run_wall_time_us":3726,"num_ops":2,"prepare.queue_time_us":1041,"prepare.run_cpu_time_us":3629,"prepare.run_wall_time_us":5997,"raft.queue_time_us":400,"raft.run_cpu_time_us":21,"raft.run_wall_time_us":24,"replication_time_us":668329,"thread_start_us":16570,"threads_started":1}]]}
I20260924 19:57:37.414904 18776 raft_consensus.cc:2272] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:37.416069 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d970cc92a1134a98a613d198a319cb01" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:37.416399 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d970cc92a1134a98a613d198a319cb01" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:37.416764 17821 raft_consensus.cc:2468] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
W20260924 19:57:37.417642 18050 leader_election.cc:343] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.418905 18051 leader_election.cc:304] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:37.420140 18746 raft_consensus.cc:2804] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:37.420463 18746 raft_consensus.cc:493] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:37.420765 18746 raft_consensus.cc:3060] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:37.430935 18223 tablet_bootstrap.cc:492] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
W20260924 19:57:37.432863 17607 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 26) took 1246 ms (client timeout 1000 ms). Trace:
W20260924 19:57:37.433301 17607 rpcz_store.cc:269] 0924 19:57:36.186297 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:36.654431 (+468134us) service_pool.cc:224] Handling call
0924 19:57:37.432787 (+778356us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11423,"reported_tablets":16}
W20260924 19:57:37.435348 17611 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 36) took 3814 ms (client timeout 1000 ms). Trace:
W20260924 19:57:37.435890 17611 rpcz_store.cc:269] 0924 19:57:33.620761 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:33.623610 (+ 2849us) service_pool.cc:224] Handling call
0924 19:57:37.435262 (+3811652us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:36.775718 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:36.776209 (+ 491us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:36.776235 (+ 26us) write_op.cc:435] Acquired schema lock
0924 19:57:36.776253 (+ 18us) tablet.cc:662] Decoding operations
0924 19:57:36.810804 (+ 34551us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:36.810855 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:36.810881 (+ 26us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:36.811065 (+ 184us) tablet.cc:701] Row locks acquired
0924 19:57:36.811078 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:57:36.811251 (+ 173us) write_op.cc:270] Start()
0924 19:57:36.811349 (+ 98us) write_op.cc:276] Timestamp: P: 1790279856811193 usec, L: 0
0924 19:57:36.811363 (+ 14us) op_driver.cc:348] REPLICATION: starting
0924 19:57:36.811907 (+ 544us) log.cc:844] Serialized 356 byte log entry
0924 19:57:36.824577 (+ 12670us) op_driver.cc:464] REPLICATION: finished
0924 19:57:36.825060 (+ 483us) write_op.cc:301] APPLY: starting
0924 19:57:36.825124 (+ 64us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:36.825236 (+ 112us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:36.825341 (+ 105us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:36.825496 (+ 155us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:36.825612 (+ 116us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:36.825667 (+ 55us) write_op.cc:312] APPLY: finished
0924 19:57:36.825846 (+ 179us) log.cc:844] Serialized 23 byte log entry
0924 19:57:36.826002 (+ 156us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:36.826064 (+ 62us) write_op.cc:454] Released schema lock
0924 19:57:36.826094 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:36.826309 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:36.826331 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":314,"apply.run_cpu_time_us":1227,"apply.run_wall_time_us":1227,"num_ops":1,"prepare.queue_time_us":129087,"prepare.run_cpu_time_us":2444,"prepare.run_wall_time_us":47247,"raft.queue_time_us":11194,"raft.run_cpu_time_us":685,"raft.run_wall_time_us":686,"replication_time_us":13114}]]}
I20260924 19:57:37.441067 18785 consensus_queue.cc:1048] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:37.447896 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f994e0401df4fc19bc5abb10f08bc6e with delete_type TABLET_DATA_DELETED (replaced by tablet e1137318a9fa4a0781f718dc34c4ecbe) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.449589 18224 raft_consensus.cc:359] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.450234 18224 raft_consensus.cc:385] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:37.450518 18224 raft_consensus.cc:740] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:37.451524 18224 consensus_queue.cc:260] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.452561 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet ef3f82de70bb4328a15a044fe0e500bf with delete_type TABLET_DATA_DELETED (replaced by tablet 57008266b80345e0a3f8e60bd7e2d62b) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.455906 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet 085b8088134d47478f0ce06a6a9f8572 with delete_type TABLET_DATA_DELETED (replaced by tablet fec9b6ecfb384b528cf71c4cfb2fd4cb) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.460384 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.462644 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet 151247d6b5254cd884c58db9311ac04a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.449855 18746 raft_consensus.cc:515] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.463354 18223 ts_tablet_manager.cc:1403] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.181s user 0.014s sys 0.002s
I20260924 19:57:37.465857 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d970cc92a1134a98a613d198a319cb01" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:37.466298 18223 raft_consensus.cc:359] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.466923 18223 raft_consensus.cc:385] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:57:37.467422 18050 leader_election.cc:343] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.481405 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet f482b53a4a2d44f6b3431c588cd605ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.487643 18777 tablet_replica.cc:333] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:37.499732 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet a2e4519aed7642678db3a518a89778de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.501930 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet a765abd702644f079bddffb3fc4238ef with delete_type TABLET_DATA_DELETED (replaced by tablet f169c47d711d4566b2004cdfd0de725d) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.504014 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d970cc92a1134a98a613d198a319cb01" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:37.504585 17821 raft_consensus.cc:3060] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:37.505256 18224 ts_tablet_manager.cc:1434] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.092s user 0.002s sys 0.005s
I20260924 19:57:37.510886 17821 raft_consensus.cc:2468] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:37.306046 18701 raft_consensus.cc:515] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.516623 18220 tablet_bootstrap.cc:492] T c1684aaf9d984649bff8c4b0b12044b9 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:37.516790 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b328d37d55e4116a29322caaa21d431" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:37.517020 18220 ts_tablet_manager.cc:1403] T c1684aaf9d984649bff8c4b0b12044b9 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.433s user 0.009s sys 0.000s
W20260924 19:57:37.518157 17753 leader_election.cc:343] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.519492 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet d970cc92a1134a98a613d198a319cb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.520104 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 41bdb846ca19425b882434df0167479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:31 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:37.507889 18223 raft_consensus.cc:740] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:37.521128 18223 consensus_queue.cc:260] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.511948 18051 leader_election.cc:304] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:37.514993 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b328d37d55e4116a29322caaa21d431" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:37.523128 18223 ts_tablet_manager.cc:1434] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.059s user 0.004s sys 0.000s
I20260924 19:57:37.523948 18117 raft_consensus.cc:3060] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:37.524979 18757 raft_consensus.cc:2804] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:37.526464 18776 ts_tablet_manager.cc:1916] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:37.531175 18117 raft_consensus.cc:2468] T 3b328d37d55e4116a29322caaa21d431 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:37.532513 17753 leader_election.cc:304] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:37.533463 18782 raft_consensus.cc:2804] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:37.534165 18782 raft_consensus.cc:697] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:37.535416 18782 consensus_queue.cc:237] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.536762 18778 consensus_queue.cc:1048] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:37.539825 18223 tablet_bootstrap.cc:492] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:37.546896 18223 tablet_bootstrap.cc:654] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:37.546896 18224 tablet_bootstrap.cc:492] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:37.548187 18777 raft_consensus.cc:2243] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:37.548776 18777 raft_consensus.cc:2272] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:37.550894 18777 ts_tablet_manager.cc:1916] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:37.551478 18746 leader_election.cc:290] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:37.552353 18746 raft_consensus.cc:493] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:37.553036 18224 tablet_bootstrap.cc:654] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:37.552772 18746 raft_consensus.cc:515] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:37.380465 17599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.189:43689, user_credentials={real_user=slave}} blocked reactor thread for 70925us
W20260924 19:57:37.370921 17612 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 36) took 808 ms (client timeout 1000 ms). Trace:
W20260924 19:57:37.374284 17615 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 37) took 2769 ms (client timeout 1000 ms). Trace:
I20260924 19:57:37.455541 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:37.567708 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:37.568554 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:37.569249 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:37.570013 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:37.570839 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet c1684aaf9d984649bff8c4b0b12044b9 (table default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe]). Expected version 1 got 0
I20260924 19:57:37.571969 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_22092e7254a9428e81176f410a135394 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 2 got 0
I20260924 19:57:37.581728 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet 41bdb846ca19425b882434df0167479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:31 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:37.582130 18089 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f994e0401df4fc19bc5abb10f08bc6e with delete_type TABLET_DATA_DELETED (replaced by tablet e1137318a9fa4a0781f718dc34c4ecbe) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:37.582497 18087 tablet_service.cc:1558] Processing DeleteTablet for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:37.583173 18092 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:37.583787 18090 tablet_service.cc:1558] Processing DeleteTablet for tablet 151247d6b5254cd884c58db9311ac04a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:37.584388 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet a2e4519aed7642678db3a518a89778de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:37.584645 17609 catalog_manager.cc:5838] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:37.586321 17609 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_22092e7254a9428e81176f410a135394 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 2 got 0
I20260924 19:57:37.587255 18757 raft_consensus.cc:697] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:37.587047 17609 catalog_manager.cc:5838] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:37.588120 18757 consensus_queue.cc:237] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.588836 18224 tablet_bootstrap.cc:492] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:37.589430 18224 ts_tablet_manager.cc:1403] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.000s
I20260924 19:57:37.591813 18224 raft_consensus.cc:359] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.592464 18224 raft_consensus.cc:385] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:37.592717 18224 raft_consensus.cc:740] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:37.593429 18224 consensus_queue.cc:260] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.595455 18224 ts_tablet_manager.cc:1434] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 19:57:37.604563 18084 tablet_service.cc:1558] Processing DeleteTablet for tablet ef3f82de70bb4328a15a044fe0e500bf with delete_type TABLET_DATA_DELETED (replaced by tablet 57008266b80345e0a3f8e60bd7e2d62b) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:37.606562 18084 tablet_service.cc:1558] Processing DeleteTablet for tablet efd7339d0fee436299a7eac02a82addd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:28 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:37.608582 17799 tablet_service.cc:1511] Processing CreateTablet for tablet bafaab10ce564b1b97659eccc0ab0967 (DEFAULT_TABLE table=default.Table_081e10d31dc74797abba37099d8a75fc [id=0ea22c5631c347aa9c7aa0e24140cb14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.609084 18084 tablet_service.cc:1511] Processing CreateTablet for tablet ca37ddd22f5d4bcab2862553408447ce (DEFAULT_TABLE table=default.Table_ca51e47e4fc44d9ba5dd07cbeefa6e68 [id=6ea0f9084fd84e848fdecdd696cdc773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.609742 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bafaab10ce564b1b97659eccc0ab0967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.610186 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca37ddd22f5d4bcab2862553408447ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.611477 17609 catalog_manager.cc:5838] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:37.612632 18701 leader_election.cc:290] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:37.613184 17797 tablet_service.cc:1511] Processing CreateTablet for tablet 57008266b80345e0a3f8e60bd7e2d62b (DEFAULT_TABLE table=default.Table_cca06980f109487a9ed8b8172950b60d [id=016f4039733144c985808b862eebcc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.614435 17797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57008266b80345e0a3f8e60bd7e2d62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.614835 18802 tablet.cc:1751] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:37.623402 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet f482b53a4a2d44f6b3431c588cd605ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:37.479902 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_ae1050d7716043fbad49a8f62505e719"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:37.403033 17614 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 38) took 1310 ms (client timeout 1000 ms). Trace:
W20260924 19:57:37.626896 17614 rpcz_store.cc:269] 0924 19:57:36.093105 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:36.650698 (+557593us) service_pool.cc:224] Handling call
0924 19:57:37.402958 (+752260us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5361,"reported_tablets":30}
I20260924 19:57:37.471743 17610 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_6c2fccb54ef24d088cd2ec174a8e20e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:37.628968 17610 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c2fccb54ef24d088cd2ec174a8e20e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:37.588531 17615 rpcz_store.cc:269] 0924 19:57:34.605437 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.605625 (+ 188us) service_pool.cc:224] Handling call
0924 19:57:37.374211 (+2768586us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:35.457478 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:35.457842 (+ 364us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:35.457865 (+ 23us) write_op.cc:435] Acquired schema lock
0924 19:57:35.457891 (+ 26us) tablet.cc:662] Decoding operations
0924 19:57:35.458139 (+ 248us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:35.458176 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:35.458197 (+ 21us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:35.458337 (+ 140us) tablet.cc:701] Row locks acquired
0924 19:57:35.458350 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:57:35.458469 (+ 119us) write_op.cc:270] Start()
0924 19:57:35.458551 (+ 82us) write_op.cc:276] Timestamp: P: 1790279855458404 usec, L: 0
0924 19:57:35.458567 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:35.459060 (+ 493us) log.cc:844] Serialized 530 byte log entry
0924 19:57:35.505292 (+ 46232us) op_driver.cc:464] REPLICATION: finished
0924 19:57:35.512246 (+ 6954us) write_op.cc:301] APPLY: starting
0924 19:57:35.512305 (+ 59us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:35.512397 (+ 92us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:35.512492 (+ 95us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:35.512634 (+ 142us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:35.512763 (+ 129us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:35.512810 (+ 47us) write_op.cc:312] APPLY: finished
0924 19:57:35.512996 (+ 186us) log.cc:844] Serialized 23 byte log entry
0924 19:57:35.513150 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:35.513220 (+ 70us) write_op.cc:454] Released schema lock
0924 19:57:35.513252 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":54014,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":6823,"apply.run_cpu_time_us":1205,"apply.run_wall_time_us":1205,"num_ops":1,"prepare.queue_time_us":27177,"prepare.run_cpu_time_us":1790,"prepare.run_wall_time_us":2864,"replication_time_us":46649,"thread_start_us":19589,"threads_started":1}]]}
W20260924 19:57:37.404827 17608 rpcz_store.cc:269] 0924 19:57:35.492404 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:35.667288 (+174884us) service_pool.cc:224] Handling call
0924 19:57:37.343908 (+1676620us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:37.252595 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:37.253020 (+ 425us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:37.253046 (+ 26us) write_op.cc:435] Acquired schema lock
0924 19:57:37.253062 (+ 16us) tablet.cc:662] Decoding operations
0924 19:57:37.253339 (+ 277us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:37.253381 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:37.253410 (+ 29us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:37.253561 (+ 151us) tablet.cc:701] Row locks acquired
0924 19:57:37.253580 (+ 19us) write_op.cc:260] PREPARE: finished
0924 19:57:37.253695 (+ 115us) write_op.cc:270] Start()
0924 19:57:37.253774 (+ 79us) write_op.cc:276] Timestamp: P: 1790279857253609 usec, L: 0
0924 19:57:37.253792 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:37.254292 (+ 500us) log.cc:844] Serialized 356 byte log entry
0924 19:57:37.310230 (+ 55938us) op_driver.cc:464] REPLICATION: finished
0924 19:57:37.310568 (+ 338us) write_op.cc:301] APPLY: starting
0924 19:57:37.310626 (+ 58us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:37.310714 (+ 88us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:37.310811 (+ 97us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:37.310977 (+ 166us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:37.311135 (+ 158us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:37.311215 (+ 80us) write_op.cc:312] APPLY: finished
0924 19:57:37.311460 (+ 245us) log.cc:844] Serialized 23 byte log entry
0924 19:57:37.311649 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:37.311723 (+ 74us) write_op.cc:454] Released schema lock
0924 19:57:37.311754 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:37.312002 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:37.312036 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":195,"apply.run_cpu_time_us":1381,"apply.run_wall_time_us":1380,"num_ops":1,"prepare.queue_time_us":100,"prepare.run_cpu_time_us":1982,"prepare.run_wall_time_us":2893,"replication_time_us":56332}]]}
I20260924 19:57:37.630946 17608 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_f40037fb3c91495e83bfe25056d5012f" } new_table_name: "default.Table_dc1e895a797842afbf6054713d1c978d" modify_external_catalogs: true
I20260924 19:57:37.630925 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 57008266b80345e0a3f8e60bd7e2d62b (DEFAULT_TABLE table=default.Table_cca06980f109487a9ed8b8172950b60d [id=016f4039733144c985808b862eebcc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.631495 18220 raft_consensus.cc:359] T c1684aaf9d984649bff8c4b0b12044b9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.631991 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57008266b80345e0a3f8e60bd7e2d62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.632166 18220 raft_consensus.cc:385] T c1684aaf9d984649bff8c4b0b12044b9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:37.632416 18220 raft_consensus.cc:740] T c1684aaf9d984649bff8c4b0b12044b9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
W20260924 19:57:37.564834 17612 rpcz_store.cc:269] 0924 19:57:36.562738 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.084493 (+521755us) service_pool.cc:224] Handling call
0924 19:57:37.370829 (+286336us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:37.211031 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:37.211515 (+ 484us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:37.211543 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:57:37.211559 (+ 16us) tablet.cc:662] Decoding operations
0924 19:57:37.211892 (+ 333us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:37.211936 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:37.211961 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:37.212129 (+ 168us) tablet.cc:701] Row locks acquired
0924 19:57:37.212146 (+ 17us) write_op.cc:260] PREPARE: finished
0924 19:57:37.212277 (+ 131us) write_op.cc:270] Start()
0924 19:57:37.212380 (+ 103us) write_op.cc:276] Timestamp: P: 1790279857212191 usec, L: 0
0924 19:57:37.212400 (+ 20us) op_driver.cc:348] REPLICATION: starting
0924 19:57:37.212958 (+ 558us) log.cc:844] Serialized 530 byte log entry
0924 19:57:37.247300 (+ 34342us) op_driver.cc:464] REPLICATION: finished
0924 19:57:37.247634 (+ 334us) write_op.cc:301] APPLY: starting
0924 19:57:37.247689 (+ 55us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:37.247784 (+ 95us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:37.247877 (+ 93us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:37.248077 (+ 200us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:37.248204 (+ 127us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:37.248245 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:57:37.248430 (+ 185us) log.cc:844] Serialized 23 byte log entry
0924 19:57:37.248593 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:37.248660 (+ 67us) write_op.cc:454] Released schema lock
0924 19:57:37.248692 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:37.248913 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:37.248935 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":174,"apply.run_cpu_time_us":1246,"apply.run_wall_time_us":1246,"num_ops":1,"prepare.queue_time_us":6012,"prepare.run_cpu_time_us":2232,"prepare.run_wall_time_us":2233,"raft.queue_time_us":1383,"raft.run_cpu_time_us":695,"raft.run_wall_time_us":695,"replication_time_us":34801}]]}
I20260924 19:57:37.634634 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_fbc5ceeb4d1242d19344825cefb6ef2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:37.598677 17611 master_service.cc:960] ReplaceTablet: received request to replace tablet 2cf62a73c49245198d65011875148dc3 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:37.635713 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c08f10d2a364cb8a0a7ea40654c3fe4" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:37.469836 17606 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51280 (request call id 190) took 2336 ms. Trace:
I20260924 19:57:37.638489 17606 rpcz_store.cc:276] 0924 19:57:35.133843 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.410664 (+2276821us) service_pool.cc:224] Handling call
0924 19:57:37.469735 (+ 59071us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:37.639406 17945 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:37.640098 17609 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:37.642982 18746 leader_election.cc:290] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:37.646275 17609 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
W20260924 19:57:37.651726 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbc5ceeb4d1242d19344825cefb6ef2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:37.653189 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae1050d7716043fbad49a8f62505e719 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:37.655402 18050 leader_election.cc:343] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.615957 17795 tablet_service.cc:1511] Processing CreateTablet for tablet ca37ddd22f5d4bcab2862553408447ce (DEFAULT_TABLE table=default.Table_ca51e47e4fc44d9ba5dd07cbeefa6e68 [id=6ea0f9084fd84e848fdecdd696cdc773]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.657996 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca37ddd22f5d4bcab2862553408447ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.667768 17941 tablet_service.cc:1511] Processing CreateTablet for tablet bafaab10ce564b1b97659eccc0ab0967 (DEFAULT_TABLE table=default.Table_081e10d31dc74797abba37099d8a75fc [id=0ea22c5631c347aa9c7aa0e24140cb14]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:57:37.669040 17600 catalog_manager.cc:5136] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): alter failed for tablet 349c6e9413bb41268d0c41d549203e2a (table default.Table_41c56bb86db34f16aba6e94147cde1d2 [id=b1d9f8b4039b4d19bfc493355e0d07cc]),no further retry: Illegal state: Tablet not RUNNING: STOPPED
I20260924 19:57:37.668612 18636 consensus_queue.cc:1048] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:37.675874 18701 raft_consensus.cc:493] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:37.676494 18701 raft_consensus.cc:515] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.673759 17941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bafaab10ce564b1b97659eccc0ab0967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.683821 18084 tablet_service.cc:1511] Processing CreateTablet for tablet bafaab10ce564b1b97659eccc0ab0967 (DEFAULT_TABLE table=default.Table_081e10d31dc74797abba37099d8a75fc [id=0ea22c5631c347aa9c7aa0e24140cb14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.685491 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bafaab10ce564b1b97659eccc0ab0967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.685971 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affce4de8e0b412aa3b4bf19f9be7b6e" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:37.672633 17945 tablet_service.cc:1511] Processing CreateTablet for tablet 57008266b80345e0a3f8e60bd7e2d62b (DEFAULT_TABLE table=default.Table_cca06980f109487a9ed8b8172950b60d [id=016f4039733144c985808b862eebcc9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.687318 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_39ac82df7ce14073912622fcd1b57aaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:37.688860 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39ac82df7ce14073912622fcd1b57aaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:37.633126 18220 consensus_queue.cc:260] T c1684aaf9d984649bff8c4b0b12044b9 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.625236 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c08f10d2a364cb8a0a7ea40654c3fe4" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:37.691977 17819 raft_consensus.cc:2468] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:37.695538 17615 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_cca06980f109487a9ed8b8172950b60d" } modify_external_catalogs: true
I20260924 19:57:37.696055 17615 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_cca06980f109487a9ed8b8172950b60d" } modify_external_catalogs: true
I20260924 19:57:37.700136 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57008266b80345e0a3f8e60bd7e2d62b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.716027 18051 leader_election.cc:304] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:37.717394 17753 leader_election.cc:343] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.738097 17946 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f994e0401df4fc19bc5abb10f08bc6e with delete_type TABLET_DATA_DELETED (replaced by tablet e1137318a9fa4a0781f718dc34c4ecbe) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:37.749137 17601 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet dbb82914df65416aaf47db8e507afb40 consensus state has no leader
I20260924 19:57:37.750156 18220 ts_tablet_manager.cc:1434] T c1684aaf9d984649bff8c4b0b12044b9 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.233s user 0.008s sys 0.000s
I20260924 19:57:37.754498 17606 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_22092e7254a9428e81176f410a135394" } new_table_name: "default.Table_7b49c63ad79c4132afbbbb9e1f2d4ca0" modify_external_catalogs: true
I20260924 19:57:37.756521 18761 tablet.cc:1746] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a: Skipping requested alter to schema version 2, tablet already version 2
I20260924 19:57:37.759042 18224 tablet_bootstrap.cc:492] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:37.763458 18782 raft_consensus.cc:493] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:37.763922 18782 raft_consensus.cc:515] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.765890 18782 leader_election.cc:290] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:37.766783 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd2f4eb8bf254c4e85b6a9072271766b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:37.767244 18220 tablet_bootstrap.cc:492] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:37.767798 17947 tablet_service.cc:1511] Processing CreateTablet for tablet d129e0979a584f40ba9a975349cf15a5 (DEFAULT_TABLE table=default.Table_fbe470367ca848268ffa88adbee30894 [id=94cea8de30ca4010a95f142ac8408653]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:57:37.768522 17753 leader_election.cc:343] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.768832 18782 raft_consensus.cc:493] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:37.768779 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d129e0979a584f40ba9a975349cf15a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.769311 18782 raft_consensus.cc:515] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.769769 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd2f4eb8bf254c4e85b6a9072271766b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:37.771181 18782 leader_election.cc:290] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
W20260924 19:57:37.771541 17753 leader_election.cc:343] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.771922 17753 leader_election.cc:304] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:37.772001 18757 raft_consensus.cc:2804] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:37.772320 18757 raft_consensus.cc:493] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:37.772617 18757 raft_consensus.cc:3060] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:37.772985 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edc6c16578e5463a9d44c25fd6c538a9" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:37.774320 18780 raft_consensus.cc:2749] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:37.775862 17949 tablet_service.cc:1511] Processing CreateTablet for tablet 1d95d6e48173436c8c59130464413ea1 (DEFAULT_TABLE table=default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.776660 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edc6c16578e5463a9d44c25fd6c538a9" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:37.778091 17753 leader_election.cc:343] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:37.778743 17753 leader_election.cc:343] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.779119 17753 leader_election.cc:304] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:37.779374 18757 raft_consensus.cc:515] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.781440 18757 leader_election.cc:290] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:37.783886 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c08f10d2a364cb8a0a7ea40654c3fe4" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:37.786423 18220 tablet_bootstrap.cc:654] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:37.792999 18782 raft_consensus.cc:2749] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:37.785411 18050 leader_election.cc:343] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.791627 18086 tablet_service.cc:1511] Processing CreateTablet for tablet d129e0979a584f40ba9a975349cf15a5 (DEFAULT_TABLE table=default.Table_fbe470367ca848268ffa88adbee30894 [id=94cea8de30ca4010a95f142ac8408653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.794548 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d129e0979a584f40ba9a975349cf15a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.799736 18701 leader_election.cc:290] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:37.792450 18084 tablet_service.cc:1511] Processing CreateTablet for tablet 1d95d6e48173436c8c59130464413ea1 (DEFAULT_TABLE table=default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.802716 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d95d6e48173436c8c59130464413ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.776853 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d95d6e48173436c8c59130464413ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.812496 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affce4de8e0b412aa3b4bf19f9be7b6e" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:37.815809 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 1d95d6e48173436c8c59130464413ea1 (DEFAULT_TABLE table=default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.816848 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d95d6e48173436c8c59130464413ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.819949 17796 tablet_service.cc:1511] Processing CreateTablet for tablet d129e0979a584f40ba9a975349cf15a5 (DEFAULT_TABLE table=default.Table_fbe470367ca848268ffa88adbee30894 [id=94cea8de30ca4010a95f142ac8408653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.820986 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d129e0979a584f40ba9a975349cf15a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.649175 17609 catalog_manager.cc:5838] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:37.822305 17609 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet c1684aaf9d984649bff8c4b0b12044b9 (table default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe]). Expected version 1 got 0
I20260924 19:57:37.827021 17800 tablet_service.cc:1511] Processing CreateTablet for tablet e1137318a9fa4a0781f718dc34c4ecbe (DEFAULT_TABLE table=default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.827886 18224 tablet_bootstrap.cc:654] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:37.828008 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1137318a9fa4a0781f718dc34c4ecbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.836023 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c08f10d2a364cb8a0a7ea40654c3fe4" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:37.836678 17819 raft_consensus.cc:3060] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:37.851545 18802 tablet.cc:1746] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202: Skipping requested alter to schema version 2, tablet already version 2
I20260924 19:57:37.861577 18224 tablet_bootstrap.cc:492] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:37.862200 18224 ts_tablet_manager.cc:1403] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.266s user 0.010s sys 0.000s
I20260924 19:57:37.872704 18746 consensus_queue.cc:1048] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:37.873262 17950 tablet_service.cc:1511] Processing CreateTablet for tablet e1137318a9fa4a0781f718dc34c4ecbe (DEFAULT_TABLE table=default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.874290 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1137318a9fa4a0781f718dc34c4ecbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.877591 18223 tablet_bootstrap.cc:492] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
W20260924 19:57:37.879654 17905 consensus_peers.cc:597] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:37.832522 18778 consensus_queue.cc:1048] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.047s
I20260924 19:57:37.866801 18224 raft_consensus.cc:359] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.883535 18224 raft_consensus.cc:385] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:37.883822 18224 raft_consensus.cc:740] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:37.884474 18224 consensus_queue.cc:260] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.885943 18224 ts_tablet_manager.cc:1434] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.023s user 0.003s sys 0.000s
I20260924 19:57:37.885928 18223 ts_tablet_manager.cc:1403] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.346s user 0.007s sys 0.005s
I20260924 19:57:37.861522 18231 ts_tablet_manager.cc:1929] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:37.891427 18231 log.cc:1199] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/349c6e9413bb41268d0c41d549203e2a
I20260924 19:57:37.892567 18231 ts_tablet_manager.cc:1950] T 349c6e9413bb41268d0c41d549203e2a P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:37.913290 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 349c6e9413bb41268d0c41d549203e2a (table default.Table_41c56bb86db34f16aba6e94147cde1d2 [id=b1d9f8b4039b4d19bfc493355e0d07cc]) successfully deleted
W20260924 19:57:37.914475 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet efd7339d0fee436299a7eac02a82addd because the tablet was not found. No further retry: Not found: Tablet not found: efd7339d0fee436299a7eac02a82addd
W20260924 19:57:37.916718 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 916877c7540a4f6685064fc2807cb140 because the tablet was not found. No further retry: Not found: Tablet not found: 916877c7540a4f6685064fc2807cb140
I20260924 19:57:37.921232 18084 tablet_service.cc:1511] Processing CreateTablet for tablet e1137318a9fa4a0781f718dc34c4ecbe (DEFAULT_TABLE table=default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:37.923193 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1137318a9fa4a0781f718dc34c4ecbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:37.924796 17609 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]). Expected version 1 got 0
I20260924 19:57:37.930315 18224 tablet_bootstrap.cc:492] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:37.901885 18231 tablet_replica.cc:333] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
W20260924 19:57:37.929450 17753 leader_election.cc:343] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.933053 17753 leader_election.cc:304] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:37.889221 18223 raft_consensus.cc:359] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.937721 18223 raft_consensus.cc:385] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:57:37.928900 17608 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6bd442f16492495f84bb8363b9bade97 consensus state has no leader
W20260924 19:57:37.934134 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 5f994e0401df4fc19bc5abb10f08bc6e because the tablet was not found. No further retry: Not found: Tablet not found: 5f994e0401df4fc19bc5abb10f08bc6e
I20260924 19:57:37.951498 17608 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 196) took 1096 ms. Trace:
W20260924 19:57:37.951629 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet ef3f82de70bb4328a15a044fe0e500bf because the tablet was not found. No further retry: Not found: Tablet not found: ef3f82de70bb4328a15a044fe0e500bf
W20260924 19:57:37.952325 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 085b8088134d47478f0ce06a6a9f8572 because the tablet was not found. No further retry: Not found: Tablet not found: 085b8088134d47478f0ce06a6a9f8572
I20260924 19:57:37.951918 17608 rpcz_store.cc:276] 0924 19:57:36.855474 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.630634 (+775160us) service_pool.cc:224] Handling call
0924 19:57:37.631619 (+ 985us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:37.632335 (+ 716us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:37.632942 (+ 607us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:37.633146 (+ 204us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:37.928354 (+295208us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:37.951436 (+ 23082us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:37.670443 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:37.670927 (+ 484us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:37.670954 (+ 27us) write_op.cc:435] Acquired schema lock
0924 19:57:37.670971 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:37.671311 (+ 340us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:37.671350 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:37.671384 (+ 34us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:37.671554 (+ 170us) tablet.cc:701] Row locks acquired
0924 19:57:37.671573 (+ 19us) write_op.cc:260] PREPARE: finished
0924 19:57:37.671698 (+ 125us) write_op.cc:270] Start()
0924 19:57:37.671788 (+ 90us) write_op.cc:276] Timestamp: P: 1790279857671616 usec, L: 0
0924 19:57:37.671805 (+ 17us) op_driver.cc:348] REPLICATION: starting
0924 19:57:37.686657 (+ 14852us) log.cc:844] Serialized 405 byte log entry
0924 19:57:37.916513 (+229856us) op_driver.cc:464] REPLICATION: finished
0924 19:57:37.926436 (+ 9923us) write_op.cc:301] APPLY: starting
0924 19:57:37.926502 (+ 66us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:37.926604 (+ 102us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:37.926727 (+ 123us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:37.926900 (+ 173us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:37.927040 (+ 140us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:37.927081 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:57:37.927307 (+ 226us) log.cc:844] Serialized 23 byte log entry
0924 19:57:37.927472 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:37.927525 (+ 53us) write_op.cc:454] Released schema lock
0924 19:57:37.927556 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":9742,"apply.run_cpu_time_us":1351,"apply.run_wall_time_us":1350,"num_ops":1,"prepare.queue_time_us":111,"prepare.run_cpu_time_us":2303,"prepare.run_wall_time_us":16490,"replication_time_us":244617,"thread_start_us":11287,"threads_started":1}]]}
I20260924 19:57:37.958082 18757 consensus_queue.cc:1048] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:37.960865 17613 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027] alter complete (version 5)
I20260924 19:57:37.967288 18757 consensus_queue.cc:1048] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:37.982595 18220 tablet_bootstrap.cc:492] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:37.937983 18223 raft_consensus.cc:740] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:37.983364 18746 raft_consensus.cc:493] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:37.984476 18223 consensus_queue.cc:260] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:37.984136 18746 raft_consensus.cc:515] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.987128 18223 ts_tablet_manager.cc:1434] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.101s user 0.003s sys 0.002s
I20260924 19:57:37.988015 18223 tablet_bootstrap.cc:492] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:37.988459 17610 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 191) took 2842 ms. Trace:
I20260924 19:57:37.990595 18220 ts_tablet_manager.cc:1403] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.224s user 0.014s sys 0.002s
I20260924 19:57:37.991601 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cf62a73c49245198d65011875148dc3" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:37.993110 18050 leader_election.cc:343] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:37.988760 17610 rpcz_store.cc:276] 0924 19:57:35.146153 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.415394 (+2269241us) service_pool.cc:224] Handling call
0924 19:57:37.471604 (+ 56210us) spinlock_profiling.cc:243] Waited 54.6 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552207166 00007fa54fdb3034 00007fa54fdb2fc2 00007fa54fdb2ef1 00007fa54fdb2e6c 00007fa54fdb2122 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:37.629427 (+157823us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:37.705518 (+ 76091us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:37.987833 (+282315us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:37.988133 (+ 300us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:37.988398 (+ 265us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:37.748643 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:37.749110 (+ 467us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:37.749136 (+ 26us) write_op.cc:435] Acquired schema lock
0924 19:57:37.749153 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:37.749468 (+ 315us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:37.749503 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:37.749528 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:37.749762 (+ 234us) tablet.cc:701] Row locks acquired
0924 19:57:37.749775 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:57:37.749908 (+ 133us) write_op.cc:270] Start()
0924 19:57:37.750004 (+ 96us) write_op.cc:276] Timestamp: P: 1790279857749810 usec, L: 0
0924 19:57:37.750020 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:37.750618 (+ 598us) log.cc:844] Serialized 433 byte log entry
0924 19:57:37.928536 (+177918us) op_driver.cc:464] REPLICATION: finished
0924 19:57:37.981512 (+ 52976us) write_op.cc:301] APPLY: starting
0924 19:57:37.982317 (+ 805us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:37.982451 (+ 134us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:37.982573 (+ 122us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:37.984476 (+ 1903us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:37.984616 (+ 140us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:37.984664 (+ 48us) write_op.cc:312] APPLY: finished
0924 19:57:37.984844 (+ 180us) log.cc:844] Serialized 29 byte log entry
0924 19:57:37.985006 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:37.985070 (+ 64us) write_op.cc:454] Released schema lock
0924 19:57:37.985103 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":125550848,"child_traces":[["op",{"apply.queue_time_us":52853,"mutex_wait_us":41746,"num_ops":2,"prepare.queue_time_us":41888,"prepare.run_cpu_time_us":2193,"prepare.run_wall_time_us":2193,"replication_time_us":178434,"thread_start_us":62523,"threads_started":1}]]}
I20260924 19:57:37.994655 17610 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_12a91a9b08e840a3bf63db9781fcf701" } modify_external_catalogs: true
I20260924 19:57:37.994848 18220 raft_consensus.cc:359] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:37.995250 17610 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_12a91a9b08e840a3bf63db9781fcf701" } modify_external_catalogs: true
I20260924 19:57:37.995586 18220 raft_consensus.cc:385] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:37.995863 18220 raft_consensus.cc:740] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:37.990293 18782 raft_consensus.cc:2749] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:38.010762 18231 raft_consensus.cc:2243] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:38.015302 18220 consensus_queue.cc:260] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:38.016242 18224 tablet_bootstrap.cc:654] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:38.017629 18220 ts_tablet_manager.cc:1434] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20260924 19:57:38.017742 18231 raft_consensus.cc:2272] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:38.020144 18778 raft_consensus.cc:493] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:38.020589 18778 raft_consensus.cc:515] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:38.023913 18778 leader_election.cc:290] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:38.025068 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c861e93ad9f40ba9ad010021e43bc2b" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:38.025403 18776 ts_tablet_manager.cc:1929] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:38.025848 17818 raft_consensus.cc:2393] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:38.025825 18776 log.cc:1199] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/349c6e9413bb41268d0c41d549203e2a
I20260924 19:57:38.026634 18776 ts_tablet_manager.cc:1950] T 349c6e9413bb41268d0c41d549203e2a P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:38.034719 18231 ts_tablet_manager.cc:1916] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:38.054126 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c861e93ad9f40ba9ad010021e43bc2b" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:38.056137 18796 consensus_queue.cc:1048] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:38.067970 18814 tablet.cc:1746] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35: Skipping requested alter to schema version 2, tablet already version 2
I20260924 19:57:38.070874 18797 consensus_queue.cc:1048] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:38.074160 18746 leader_election.cc:290] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:38.078961 18223 tablet_bootstrap.cc:654] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:38.098935 18796 consensus_queue.cc:1048] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:38.109036 17611 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 2cf62a73c49245198d65011875148dc3
I20260924 19:57:38.111039 17611 catalog_manager.cc:6934] ReplaceTablet: tablet 2cf62a73c49245198d65011875148dc3 deleted and replaced by tablet cfe0303596204e528585c4fe1c476d8b
I20260924 19:57:38.020172 18220 tablet_bootstrap.cc:492] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:38.111461 17611 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51280 (request call id 193) took 2339 ms. Trace:
I20260924 19:57:38.113415 18776 tablet_replica.cc:333] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:38.113953 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet 2cf62a73c49245198d65011875148dc3 with delete_type TABLET_DATA_DELETED (replaced by tablet cfe0303596204e528585c4fe1c476d8b) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:38.111896 17611 rpcz_store.cc:276] 0924 19:57:35.772209 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.598556 (+1826347us) service_pool.cc:224] Handling call
0924 19:57:38.111410 (+512854us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:38.037515 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:38.038405 (+ 890us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:38.038468 (+ 63us) write_op.cc:435] Acquired schema lock
0924 19:57:38.038516 (+ 48us) tablet.cc:662] Decoding operations
0924 19:57:38.039227 (+ 711us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:38.039338 (+ 111us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:38.039362 (+ 24us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:38.039907 (+ 545us) tablet.cc:701] Row locks acquired
0924 19:57:38.039954 (+ 47us) write_op.cc:260] PREPARE: finished
0924 19:57:38.040581 (+ 627us) write_op.cc:270] Start()
0924 19:57:38.040785 (+ 204us) write_op.cc:276] Timestamp: P: 1790279858040428 usec, L: 0
0924 19:57:38.040807 (+ 22us) op_driver.cc:348] REPLICATION: starting
0924 19:57:38.041928 (+ 1121us) log.cc:844] Serialized 625 byte log entry
0924 19:57:38.085230 (+ 43302us) op_driver.cc:464] REPLICATION: finished
0924 19:57:38.085529 (+ 299us) write_op.cc:301] APPLY: starting
0924 19:57:38.085584 (+ 55us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:38.085705 (+ 121us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:38.085805 (+ 100us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:38.086340 (+ 535us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:38.086596 (+ 256us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:38.086657 (+ 61us) write_op.cc:312] APPLY: finished
0924 19:57:38.086895 (+ 238us) log.cc:844] Serialized 29 byte log entry
0924 19:57:38.087060 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.087141 (+ 81us) write_op.cc:454] Released schema lock
0924 19:57:38.087227 (+ 86us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:38.087467 (+ 240us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.087498 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":152,"apply.run_cpu_time_us":1770,"apply.run_wall_time_us":1909,"num_ops":2,"prepare.queue_time_us":275077,"prepare.run_cpu_time_us":3340,"prepare.run_wall_time_us":44850,"replication_time_us":44334}]]}
I20260924 19:57:38.111042 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cf62a73c49245198d65011875148dc3" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:38.116230 17614 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 194) took 2245 ms. Trace:
I20260924 19:57:38.116593 17821 raft_consensus.cc:2468] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
W20260924 19:57:38.117151 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 55133.1us
I20260924 19:57:38.117493 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 349c6e9413bb41268d0c41d549203e2a (table default.Table_41c56bb86db34f16aba6e94147cde1d2 [id=b1d9f8b4039b4d19bfc493355e0d07cc]) successfully deleted
W20260924 19:57:38.118167 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet efd7339d0fee436299a7eac02a82addd because the tablet was not found. No further retry: Not found: Tablet not found: efd7339d0fee436299a7eac02a82addd
I20260924 19:57:38.119587 17615 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 57008266b80345e0a3f8e60bd7e2d62b
I20260924 19:57:38.122642 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet 57008266b80345e0a3f8e60bd7e2d62b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:37 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:38.123009 17615 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 195) took 2013 ms. Trace:
I20260924 19:57:38.125850 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 2cf62a73c49245198d65011875148dc3 with delete_type TABLET_DATA_DELETED (replaced by tablet cfe0303596204e528585c4fe1c476d8b) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:38.126102 17607 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 192) took 2541 ms. Trace:
I20260924 19:57:38.126899 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 57008266b80345e0a3f8e60bd7e2d62b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:37 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:38.126430 17607 rpcz_store.cc:276] 0924 19:57:35.584680 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.434357 (+1849677us) service_pool.cc:224] Handling call
0924 19:57:37.479798 (+ 45441us) spinlock_profiling.cc:243] Waited 44.5 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552207166 00007fa54fdb3034 00007fa54fdb2fc2 00007fa54fdb2ef1 00007fa54fdb2e6c 00007fa54fdb2122 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:37.653857 (+174059us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:37.708161 (+ 54304us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:38.009537 (+301376us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:38.125738 (+116201us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:38.126048 (+ 310us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:37.751547 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:37.751998 (+ 451us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:37.752025 (+ 27us) write_op.cc:435] Acquired schema lock
0924 19:57:37.752042 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:37.752350 (+ 308us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:37.752396 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:37.752426 (+ 30us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:37.752657 (+ 231us) tablet.cc:701] Row locks acquired
0924 19:57:37.752675 (+ 18us) write_op.cc:260] PREPARE: finished
0924 19:57:37.752793 (+ 118us) write_op.cc:270] Start()
0924 19:57:37.752871 (+ 78us) write_op.cc:276] Timestamp: P: 1790279857752704 usec, L: 0
0924 19:57:37.752894 (+ 23us) op_driver.cc:348] REPLICATION: starting
0924 19:57:37.753420 (+ 526us) log.cc:844] Serialized 433 byte log entry
0924 19:57:37.991385 (+237965us) op_driver.cc:464] REPLICATION: finished
0924 19:57:38.007770 (+ 16385us) write_op.cc:301] APPLY: starting
0924 19:57:38.007836 (+ 66us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:38.008039 (+ 203us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:38.008164 (+ 125us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:38.008661 (+ 497us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:38.008819 (+ 158us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:38.008863 (+ 44us) write_op.cc:312] APPLY: finished
0924 19:57:38.009072 (+ 209us) log.cc:844] Serialized 29 byte log entry
0924 19:57:38.009232 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.009298 (+ 66us) write_op.cc:454] Released schema lock
0924 19:57:38.009330 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:38.010007 (+ 677us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.010059 (+ 52us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":102339968,"child_traces":[["op",{"apply.queue_time_us":16188,"apply.run_cpu_time_us":1773,"apply.run_wall_time_us":2174,"mutex_wait_us":41679,"num_ops":2,"prepare.queue_time_us":42226,"prepare.run_cpu_time_us":2095,"prepare.run_wall_time_us":2095,"replication_time_us":238383}]]}
I20260924 19:57:38.133706 18223 tablet_bootstrap.cc:492] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:38.134133 18223 ts_tablet_manager.cc:1403] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.146s user 0.004s sys 0.010s
I20260924 19:57:38.124495 17950 tablet_service.cc:1558] Processing DeleteTablet for tablet 57008266b80345e0a3f8e60bd7e2d62b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:37 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:38.125041 17949 tablet_service.cc:1558] Processing DeleteTablet for tablet 2cf62a73c49245198d65011875148dc3 with delete_type TABLET_DATA_DELETED (replaced by tablet cfe0303596204e528585c4fe1c476d8b) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:38.140524 18776 raft_consensus.cc:2243] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:38.105587 17819 raft_consensus.cc:2468] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:38.142241 18776 raft_consensus.cc:2272] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:38.144447 18776 ts_tablet_manager.cc:1916] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:38.149560 18797 consensus_queue.cc:1048] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:38.151593 18051 leader_election.cc:304] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:38.152760 18797 raft_consensus.cc:2804] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:38.153195 18797 raft_consensus.cc:493] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:38.153673 18797 raft_consensus.cc:3060] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:38.066144 17905 leader_election.cc:304] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:38.158876 18778 raft_consensus.cc:3060] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:38.162122 18797 raft_consensus.cc:515] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:38.158317 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 92270.3us
I20260924 19:57:38.167379 18761 tablet.cc:1751] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:38.129879 18220 tablet_bootstrap.cc:654] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:38.179013 18824 consensus_queue.cc:1048] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:38.185467 18224 tablet_bootstrap.cc:492] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:38.187184 18051 leader_election.cc:304] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:38.189040 18224 ts_tablet_manager.cc:1403] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.259s user 0.013s sys 0.005s
I20260924 19:57:38.192675 18797 leader_election.cc:290] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:38.195384 18785 consensus_queue.cc:1048] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:38.196085 18224 raft_consensus.cc:359] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:38.196738 18224 raft_consensus.cc:385] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:38.196996 18224 raft_consensus.cc:740] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:38.197564 18224 consensus_queue.cc:260] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:38.201454 18778 raft_consensus.cc:2749] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:38.203375 18746 raft_consensus.cc:2804] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:38.203814 18746 raft_consensus.cc:697] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:38.204643 18746 consensus_queue.cc:237] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:38.206862 18231 ts_tablet_manager.cc:1929] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:38.208372 17972 raft_consensus.cc:3060] T c1684aaf9d984649bff8c4b0b12044b9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:38.116585 17614 rpcz_store.cc:276] 0924 19:57:35.870665 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.627817 (+1757152us) service_pool.cc:224] Handling call
0924 19:57:37.689224 (+ 61407us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:37.690900 (+ 1676us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:38.015319 (+324419us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:38.115898 (+100579us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:38.116178 (+ 280us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:37.745273 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:37.745778 (+ 505us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:37.745809 (+ 31us) write_op.cc:435] Acquired schema lock
0924 19:57:37.745828 (+ 19us) tablet.cc:662] Decoding operations
0924 19:57:37.746140 (+ 312us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:37.746190 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:37.746216 (+ 26us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:37.746466 (+ 250us) tablet.cc:701] Row locks acquired
0924 19:57:37.746478 (+ 12us) write_op.cc:260] PREPARE: finished
0924 19:57:37.746609 (+ 131us) write_op.cc:270] Start()
0924 19:57:37.746706 (+ 97us) write_op.cc:276] Timestamp: P: 1790279857746514 usec, L: 0
0924 19:57:37.746724 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:37.747312 (+ 588us) log.cc:844] Serialized 433 byte log entry
0924 19:57:37.928289 (+180977us) op_driver.cc:464] REPLICATION: finished
0924 19:57:37.977997 (+ 49708us) write_op.cc:301] APPLY: starting
0924 19:57:37.978059 (+ 62us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:37.978177 (+ 118us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:37.978286 (+ 109us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:37.978765 (+ 479us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:37.978954 (+ 189us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:37.979018 (+ 64us) write_op.cc:312] APPLY: finished
0924 19:57:37.979289 (+ 271us) log.cc:844] Serialized 29 byte log entry
0924 19:57:37.979480 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:37.979569 (+ 89us) write_op.cc:454] Released schema lock
0924 19:57:37.979614 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:38.028116 (+ 48502us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.028145 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":49505,"apply.run_cpu_time_us":1948,"apply.run_wall_time_us":50082,"num_ops":2,"prepare.queue_time_us":45178,"prepare.run_cpu_time_us":2278,"prepare.run_wall_time_us":2278,"replication_time_us":181475}]]}
I20260924 19:57:38.210374 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cf62a73c49245198d65011875148dc3" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:38.223412 17970 raft_consensus.cc:3060] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:38.123368 17615 rpcz_store.cc:276] 0924 19:57:36.110102 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.672613 (+1562511us) service_pool.cc:224] Handling call
0924 19:57:37.696458 (+ 23845us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:37.698071 (+ 1613us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:37.698191 (+ 120us) catalog_manager.cc:2863] Locking tablets
0924 19:57:37.698845 (+ 654us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:38.043279 (+344434us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:38.119411 (+ 76132us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:38.119507 (+ 96us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:38.122889 (+ 3382us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:37.762794 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:37.899414 (+136620us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:37.899847 (+ 433us) write_op.cc:435] Acquired schema lock
0924 19:57:37.901434 (+ 1587us) tablet.cc:662] Decoding operations
0924 19:57:37.907548 (+ 6114us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:37.907609 (+ 61us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:37.907647 (+ 38us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:37.908124 (+ 477us) tablet.cc:701] Row locks acquired
0924 19:57:37.908154 (+ 30us) write_op.cc:260] PREPARE: finished
0924 19:57:37.908297 (+ 143us) write_op.cc:270] Start()
0924 19:57:37.908399 (+ 102us) write_op.cc:276] Timestamp: P: 1790279857908229 usec, L: 0
0924 19:57:37.908420 (+ 21us) op_driver.cc:348] REPLICATION: starting
0924 19:57:38.002785 (+ 94365us) log.cc:844] Serialized 728 byte log entry
0924 19:57:38.004261 (+ 1476us) spinlock_profiling.cc:243] Waited 93.8 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f7aa1 00007fa55038a5e8 00007fa55081a741 00007fa550819f18 00007fa55081ebb1 00007fa55081eb69 00007fa55081eaf9 00007fa55081eac1 00007fa55081dced 00000000005bd4e4 00000000005bd319
0924 19:57:38.040687 (+ 36426us) op_driver.cc:464] REPLICATION: finished
0924 19:57:38.042016 (+ 1329us) write_op.cc:301] APPLY: starting
0924 19:57:38.042072 (+ 56us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:38.042191 (+ 119us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:38.042208 (+ 17us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:38.042493 (+ 285us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:38.042628 (+ 135us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:38.042674 (+ 46us) write_op.cc:312] APPLY: finished
0924 19:57:38.042850 (+ 176us) log.cc:844] Serialized 29 byte log entry
0924 19:57:38.042977 (+ 127us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.043041 (+ 64us) write_op.cc:454] Released schema lock
0924 19:57:38.043074 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:38.043552 (+ 478us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.043604 (+ 52us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1163,"apply.run_cpu_time_us":1306,"apply.run_wall_time_us":1476,"mutex_wait_us":11015,"num_ops":2,"prepare.queue_time_us":17948,"prepare.run_cpu_time_us":5364,"prepare.run_wall_time_us":274308,"raft.queue_time_us":6028,"raft.run_cpu_time_us":951,"raft.run_wall_time_us":1239,"replication_time_us":132175,"spinlock_wait_cycles":215934464}]]}
I20260924 19:57:38.233515 17970 raft_consensus.cc:2468] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:38.238466 18746 consensus_queue.cc:1048] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:38.241106 18050 leader_election.cc:304] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:38.242110 18824 raft_consensus.cc:2804] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:38.242563 18824 raft_consensus.cc:697] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:38.243026 18814 tablet.cc:1751] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:38.242137 18746 consensus_queue.cc:1048] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260924 19:57:38.243392 18824 consensus_queue.cc:237] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:38.218389 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2cf62a73c49245198d65011875148dc3" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:38.271499 17821 raft_consensus.cc:3060] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:38.277979 18231 log.cc:1199] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:38.283995 18231 ts_tablet_manager.cc:1950] T 24f84032bca24a7ca9da65f0bbceb0d5 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:38.286662 18220 tablet_bootstrap.cc:492] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:38.287075 18220 ts_tablet_manager.cc:1403] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.267s user 0.009s sys 0.003s
I20260924 19:57:38.289523 18220 raft_consensus.cc:359] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:38.290153 18220 raft_consensus.cc:385] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:38.313553 17612 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 197) took 1457 ms. Trace:
I20260924 19:57:38.313938 17612 rpcz_store.cc:276] 0924 19:57:36.856304 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.634251 (+777947us) service_pool.cc:224] Handling call
0924 19:57:37.652209 (+ 17958us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:37.685674 (+ 33465us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:38.312916 (+627242us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:38.313215 (+ 299us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:38.313492 (+ 277us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:38.082433 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:38.082898 (+ 465us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:38.082927 (+ 29us) write_op.cc:435] Acquired schema lock
0924 19:57:38.082938 (+ 11us) tablet.cc:662] Decoding operations
0924 19:57:38.083275 (+ 337us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:38.083312 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:38.083333 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:38.083574 (+ 241us) tablet.cc:701] Row locks acquired
0924 19:57:38.083591 (+ 17us) write_op.cc:260] PREPARE: finished
0924 19:57:38.083715 (+ 124us) write_op.cc:270] Start()
0924 19:57:38.083805 (+ 90us) write_op.cc:276] Timestamp: P: 1790279858083635 usec, L: 0
0924 19:57:38.083822 (+ 17us) op_driver.cc:348] REPLICATION: starting
0924 19:57:38.084385 (+ 563us) log.cc:844] Serialized 433 byte log entry
0924 19:57:38.306878 (+222493us) op_driver.cc:464] REPLICATION: finished
0924 19:57:38.309680 (+ 2802us) write_op.cc:301] APPLY: starting
0924 19:57:38.309748 (+ 68us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:38.309865 (+ 117us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:38.309959 (+ 94us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:38.310384 (+ 425us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:38.310534 (+ 150us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:38.310582 (+ 48us) write_op.cc:312] APPLY: finished
0924 19:57:38.310771 (+ 189us) log.cc:844] Serialized 29 byte log entry
0924 19:57:38.310944 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.311016 (+ 72us) write_op.cc:454] Released schema lock
0924 19:57:38.311050 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:38.311321 (+ 271us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.311357 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2625,"apply.run_cpu_time_us":1610,"apply.run_wall_time_us":1609,"num_ops":2,"prepare.queue_time_us":205907,"prepare.run_cpu_time_us":2191,"prepare.run_wall_time_us":2191,"replication_time_us":222925}]]}
I20260924 19:57:38.346123 18802 tablet.cc:1751] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:38.349188 17821 raft_consensus.cc:2468] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:38.353261 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 24f84032bca24a7ca9da65f0bbceb0d5 (table default.Table_5f33677cfb6a4eab8497cae637ecdf26 [id=6e11b13ce19a4d12893da7865efea4b4]) successfully deleted
I20260924 19:57:38.355929 18776 ts_tablet_manager.cc:1929] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:38.356360 18776 log.cc:1199] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/a2e4519aed7642678db3a518a89778de
I20260924 19:57:38.357120 18776 ts_tablet_manager.cc:1950] T a2e4519aed7642678db3a518a89778de P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:38.365796 18701 raft_consensus.cc:493] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:38.366353 18701 raft_consensus.cc:515] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:38.366926 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 151247d6b5254cd884c58db9311ac04a because the tablet was not found. No further retry: Not found: Tablet not found: 151247d6b5254cd884c58db9311ac04a
I20260924 19:57:38.369376 18701 leader_election.cc:290] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:38.380561 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet f482b53a4a2d44f6b3431c588cd605ce because the tablet was not found. No further retry: Not found: Tablet not found: f482b53a4a2d44f6b3431c588cd605ce
W20260924 19:57:38.382216 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet a2e4519aed7642678db3a518a89778de because the tablet was not found. No further retry: Not found: Tablet not found: a2e4519aed7642678db3a518a89778de
I20260924 19:57:38.383693 18777 ts_tablet_manager.cc:1929] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 19:57:38.383924 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet a765abd702644f079bddffb3fc4238ef because the tablet was not found. No further retry: Not found: Tablet not found: a765abd702644f079bddffb3fc4238ef
I20260924 19:57:38.384123 18777 log.cc:1199] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/349c6e9413bb41268d0c41d549203e2a
I20260924 19:57:38.384927 18777 ts_tablet_manager.cc:1950] T 349c6e9413bb41268d0c41d549203e2a P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:38.385753 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 149) took 1166 ms. Trace:
I20260924 19:57:38.386089 18777 rpcz_store.cc:276] 0924 19:57:37.219724 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.233511 (+ 13787us) service_pool.cc:224] Handling call
0924 19:57:37.487471 (+253960us) ts_tablet_manager.cc:1151] Deleting tablet 349c6e9413bb41268d0c41d549203e2a
0924 19:57:37.487519 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:37.919629 (+432110us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:38.383544 (+463915us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:38.385694 (+ 2150us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":253373,"thread_start_us":254991,"threads_started":1}
W20260924 19:57:38.388121 17906 consensus_peers.cc:597] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:38.391575 17614 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_56ddb56503844349ba42c68ebb67a694" } new_table_name: "default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4" modify_external_catalogs: true
I20260924 19:57:38.395814 18224 ts_tablet_manager.cc:1434] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.202s user 0.015s sys 0.002s
I20260924 19:57:38.397135 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_97a3c9fb681344628ba099ed831d60b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:38.398732 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97a3c9fb681344628ba099ed831d60b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:38.406158 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0246d83189d4ba5b75fbd8390f517f6" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:38.407311 18777 tablet_replica.cc:333] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:38.408540 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_c449d7c2b02a4bcdb7091a3560bcaa02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:38.410034 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c449d7c2b02a4bcdb7091a3560bcaa02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:38.411478 18776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 148) took 1156 ms. Trace:
I20260924 19:57:38.411872 18776 rpcz_store.cc:276] 0924 19:57:37.255299 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.256712 (+ 1413us) service_pool.cc:224] Handling call
0924 19:57:38.113235 (+856523us) ts_tablet_manager.cc:1151] Deleting tablet a2e4519aed7642678db3a518a89778de
0924 19:57:38.113284 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.238133 (+124849us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:38.355786 (+117653us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:38.411406 (+ 55620us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":856021}
I20260924 19:57:38.412297 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0246d83189d4ba5b75fbd8390f517f6" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:38.413372 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet a2e4519aed7642678db3a518a89778de (table default.Table_a4745efada6f4edea3f2d4bad1c3905e [id=993f4ef55cb44fb09a9c543e7cf1f2b8]) successfully deleted
I20260924 19:57:38.422188 18777 raft_consensus.cc:2243] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:38.423038 18777 raft_consensus.cc:2272] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:38.290432 18220 raft_consensus.cc:740] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:38.428001 18231 tablet_replica.cc:333] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:38.429749 18224 tablet_bootstrap.cc:492] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:38.430940 17753 leader_election.cc:343] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:38.431264 18839 raft_consensus.cc:493] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:38.433015 18839 raft_consensus.cc:515] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:38.435263 18777 ts_tablet_manager.cc:1916] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:38.444242 18839 leader_election.cc:290] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:38.429566 18776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 149) took 1174 ms. Trace:
I20260924 19:57:38.447328 18776 rpcz_store.cc:276] 0924 19:57:37.255557 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.257289 (+ 1732us) service_pool.cc:224] Handling call
0924 19:57:38.429250 (+1171961us) ts_tablet_manager.cc:1151] Deleting tablet ef3f82de70bb4328a15a044fe0e500bf
0924 19:57:38.429301 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.429495 (+ 194us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171440}
I20260924 19:57:38.448253 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 349c6e9413bb41268d0c41d549203e2a (table default.Table_41c56bb86db34f16aba6e94147cde1d2 [id=b1d9f8b4039b4d19bfc493355e0d07cc]) successfully deleted
I20260924 19:57:38.449023 18776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 150) took 1193 ms. Trace:
I20260924 19:57:38.136534 18223 raft_consensus.cc:359] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:38.457839 18223 raft_consensus.cc:385] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:38.458088 18223 raft_consensus.cc:740] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
W20260924 19:57:38.459069 17753 leader_election.cc:343] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:38.458788 18223 consensus_queue.cc:260] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:38.463984 17753 leader_election.cc:304] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:38.446924 18220 consensus_queue.cc:260] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:38.468295 18839 raft_consensus.cc:2749] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:38.471529 18220 ts_tablet_manager.cc:1434] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.184s user 0.006s sys 0.000s
I20260924 19:57:38.473726 18220 tablet_bootstrap.cc:492] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:38.474131 18839 consensus_queue.cc:1048] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:38.479578 17608 catalog_manager.cc:5838] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:38.481233 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:38.482112 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6bd442f16492495f84bb8363b9bade97" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:38.482826 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6bd442f16492495f84bb8363b9bade97" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:38.459123 18776 rpcz_store.cc:276] 0924 19:57:37.255857 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.257891 (+ 2034us) service_pool.cc:224] Handling call
0924 19:57:38.448737 (+1190846us) ts_tablet_manager.cc:1151] Deleting tablet 9c2440e1cade4327a63d6234e3adcd64
0924 19:57:38.448784 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.448968 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1190361}
I20260924 19:57:38.493593 18776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 152) took 1225 ms. Trace:
I20260924 19:57:38.494400 18223 ts_tablet_manager.cc:1434] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.360s user 0.005s sys 0.001s
W20260924 19:57:38.494505 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 9c2440e1cade4327a63d6234e3adcd64 because the tablet was not found. No further retry: Not found: Tablet not found: 9c2440e1cade4327a63d6234e3adcd64
I20260924 19:57:38.494177 18776 rpcz_store.cc:276] 0924 19:57:37.268218 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.268336 (+ 118us) service_pool.cc:224] Handling call
0924 19:57:38.493254 (+1224918us) ts_tablet_manager.cc:1151] Deleting tablet 151247d6b5254cd884c58db9311ac04a
0924 19:57:38.493302 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.493515 (+ 213us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1224255}
I20260924 19:57:38.495550 18223 tablet_bootstrap.cc:492] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:38.496333 18776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 154) took 1226 ms. Trace:
W20260924 19:57:38.496502 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 151247d6b5254cd884c58db9311ac04a because the tablet was not found. No further retry: Not found: Tablet not found: 151247d6b5254cd884c58db9311ac04a
W20260924 19:57:38.500764 17753 leader_election.cc:343] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:38.501408 17753 leader_election.cc:343] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:38.501777 17753 leader_election.cc:304] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:38.502619 18231 raft_consensus.cc:2243] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:38.507748 18839 raft_consensus.cc:2749] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:38.530776 18231 raft_consensus.cc:2272] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:38.531257 18776 rpcz_store.cc:276] 0924 19:57:37.269708 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.271848 (+ 2140us) service_pool.cc:224] Handling call
0924 19:57:38.496063 (+1224215us) ts_tablet_manager.cc:1151] Deleting tablet 085b8088134d47478f0ce06a6a9f8572
0924 19:57:38.496108 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.496275 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1223685}
I20260924 19:57:38.532721 18231 ts_tablet_manager.cc:1916] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:38.532725 18776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 155) took 1260 ms. Trace:
W20260924 19:57:38.532902 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 085b8088134d47478f0ce06a6a9f8572 because the tablet was not found. No further retry: Not found: Tablet not found: 085b8088134d47478f0ce06a6a9f8572
W20260924 19:57:38.529165 17753 consensus_peers.cc:597] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:38.533129 18776 rpcz_store.cc:276] 0924 19:57:37.272772 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.272866 (+ 94us) service_pool.cc:224] Handling call
0924 19:57:38.532453 (+1259587us) ts_tablet_manager.cc:1151] Deleting tablet f482b53a4a2d44f6b3431c588cd605ce
0924 19:57:38.532500 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.532667 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1259106}
W20260924 19:57:38.534648 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet f482b53a4a2d44f6b3431c588cd605ce because the tablet was not found. No further retry: Not found: Tablet not found: f482b53a4a2d44f6b3431c588cd605ce
I20260924 19:57:38.535745 18776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 151) took 1268 ms. Trace:
I20260924 19:57:38.538769 18220 tablet_bootstrap.cc:654] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:38.541775 18223 tablet_bootstrap.cc:654] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:38.562556 18223 tablet_bootstrap.cc:492] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:38.565244 18223 ts_tablet_manager.cc:1403] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.002s
I20260924 19:57:38.574440 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_e91b551877474bb0914b854a58447b3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:38.536146 18776 rpcz_store.cc:276] 0924 19:57:37.267865 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.274019 (+ 6154us) service_pool.cc:224] Handling call
0924 19:57:38.535481 (+1261462us) ts_tablet_manager.cc:1151] Deleting tablet 916877c7540a4f6685064fc2807cb140
0924 19:57:38.535525 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.535692 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1260896}
W20260924 19:57:38.576457 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e91b551877474bb0914b854a58447b3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:38.576575 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 916877c7540a4f6685064fc2807cb140 because the tablet was not found. No further retry: Not found: Tablet not found: 916877c7540a4f6685064fc2807cb140
I20260924 19:57:38.483104 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:38.568616 18223 raft_consensus.cc:359] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:38.586313 18223 raft_consensus.cc:385] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:38.587393 18223 raft_consensus.cc:740] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:38.589274 18223 consensus_queue.cc:260] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:38.602438 17608 catalog_manager.cc:5838] T fd6e656a653a43d1843a91fee813237d P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:38.603751 18776 tablet_replica.cc:333] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:38.604374 17608 catalog_manager.cc:5838] T 6b7f2dd8ba2a4cb190e9ff92a542bacb P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:38.605013 18776 raft_consensus.cc:2243] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:38.609817 18776 raft_consensus.cc:2272] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:38.610954 18814 tablet.cc:1751] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:38.622926 18802 tablet.cc:1751] T c1684aaf9d984649bff8c4b0b12044b9 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:38.626067 18776 ts_tablet_manager.cc:1916] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:38.633491 17607 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_5b3a0213ccbb4622a13a68e24d12af4e" } modify_external_catalogs: true
I20260924 19:57:38.635385 17607 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_5b3a0213ccbb4622a13a68e24d12af4e" } modify_external_catalogs: true
I20260924 19:57:38.637063 18224 tablet_bootstrap.cc:654] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:38.640005 18777 ts_tablet_manager.cc:1929] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:38.640606 18777 log.cc:1199] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/41bdb846ca19425b882434df0167479e
I20260924 19:57:38.641741 18777 ts_tablet_manager.cc:1950] T 41bdb846ca19425b882434df0167479e P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:38.634215 18761 tablet.cc:1751] T c1684aaf9d984649bff8c4b0b12044b9 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260924 19:57:38.656488 18051 consensus_peers.cc:597] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:38.663271 18223 ts_tablet_manager.cc:1434] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.097s user 0.006s sys 0.000s
I20260924 19:57:38.667665 18220 tablet_bootstrap.cc:492] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:38.668143 18220 ts_tablet_manager.cc:1403] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.195s user 0.015s sys 0.004s
I20260924 19:57:38.668627 18223 tablet_bootstrap.cc:492] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:38.670439 17602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.190:46309, user_credentials={real_user=slave}} blocked reactor thread for 59060.5us
I20260924 19:57:38.673058 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 155) took 1091 ms. Trace:
I20260924 19:57:38.673576 18777 rpcz_store.cc:276] 0924 19:57:37.581416 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.581563 (+ 147us) service_pool.cc:224] Handling call
0924 19:57:38.407126 (+825563us) ts_tablet_manager.cc:1151] Deleting tablet 41bdb846ca19425b882434df0167479e
0924 19:57:38.407195 (+ 69us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.544635 (+137440us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:38.639797 (+ 95162us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:38.672967 (+ 33170us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":824860}
I20260924 19:57:38.675447 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 41bdb846ca19425b882434df0167479e (table default.Table_0bb01b569e3c41a4a53d6eb7817aaade [id=ad9f72129d8343e0932f8035a6554aab]) successfully deleted
I20260924 19:57:38.679567 18223 tablet_bootstrap.cc:654] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:38.696475 18223 tablet_bootstrap.cc:492] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:38.696913 18223 ts_tablet_manager.cc:1403] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.001s
I20260924 19:57:38.704552 18223 raft_consensus.cc:359] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:38.705265 18223 raft_consensus.cc:385] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:38.708887 18223 raft_consensus.cc:740] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:38.710363 18223 consensus_queue.cc:260] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:38.714812 18220 raft_consensus.cc:359] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:38.716800 18223 ts_tablet_manager.cc:1434] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
W20260924 19:57:38.724764 18050 consensus_peers.cc:597] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:38.725485 18746 raft_consensus.cc:493] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:38.726025 18746 raft_consensus.cc:515] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:38.731066 18746 leader_election.cc:290] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:38.777833 18224 tablet_bootstrap.cc:492] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:38.778558 18224 ts_tablet_manager.cc:1403] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.350s user 0.007s sys 0.005s
I20260924 19:57:38.747565 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbb82914df65416aaf47db8e507afb40" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:38.792654 18050 leader_election.cc:343] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:38.796555 18220 raft_consensus.cc:385] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:38.797030 18220 raft_consensus.cc:740] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:38.798092 18220 consensus_queue.cc:260] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:38.814065 17600 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet edc6c16578e5463a9d44c25fd6c538a9 consensus state has no leader
I20260924 19:57:38.832360 18223 tablet_bootstrap.cc:492] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:38.832389 17599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.189:43689, user_credentials={real_user=slave}} blocked reactor thread for 128952us
I20260924 19:57:38.839344 18776 ts_tablet_manager.cc:1929] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:38.841123 18223 tablet_bootstrap.cc:654] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:38.842252 18776 log.cc:1199] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:38.847988 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 157) took 1266 ms. Trace:
I20260924 19:57:38.848343 18777 rpcz_store.cc:276] 0924 19:57:37.581870 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.581982 (+ 112us) service_pool.cc:224] Handling call
0924 19:57:38.719988 (+1138006us) ts_tablet_manager.cc:1151] Deleting tablet 5f994e0401df4fc19bc5abb10f08bc6e
0924 19:57:38.847759 (+127771us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.847924 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1137354}
I20260924 19:57:38.849530 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 156) took 1267 ms. Trace:
W20260924 19:57:38.849669 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 5f994e0401df4fc19bc5abb10f08bc6e because the tablet was not found. No further retry: Not found: Tablet not found: 5f994e0401df4fc19bc5abb10f08bc6e
I20260924 19:57:38.849843 18777 rpcz_store.cc:276] 0924 19:57:37.581648 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.582352 (+ 704us) service_pool.cc:224] Handling call
0924 19:57:38.849273 (+1266921us) ts_tablet_manager.cc:1151] Deleting tablet fec9b6ecfb384b528cf71c4cfb2fd4cb
0924 19:57:38.849317 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.849480 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1266370}
I20260924 19:57:38.850967 18777 tablet_replica.cc:333] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
W20260924 19:57:38.851207 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb because the tablet was not found. No further retry: Not found: Tablet not found: fec9b6ecfb384b528cf71c4cfb2fd4cb
I20260924 19:57:38.863745 18839 raft_consensus.cc:493] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:38.864300 18839 raft_consensus.cc:515] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:38.866968 18777 raft_consensus.cc:2243] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:38.867689 18777 raft_consensus.cc:2272] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:38.870090 18777 ts_tablet_manager.cc:1916] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:38.883608 18776 ts_tablet_manager.cc:1950] T 24f84032bca24a7ca9da65f0bbceb0d5 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:38.890287 18776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 158) took 1251 ms. Trace:
I20260924 19:57:38.895231 18776 rpcz_store.cc:276] 0924 19:57:37.638883 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.639270 (+ 387us) service_pool.cc:224] Handling call
0924 19:57:38.603491 (+964221us) ts_tablet_manager.cc:1151] Deleting tablet 24f84032bca24a7ca9da65f0bbceb0d5
0924 19:57:38.603565 (+ 74us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.760029 (+156464us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:38.803040 (+ 43011us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:38.890219 (+ 87179us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":963645}
I20260924 19:57:38.900904 18776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 157) took 1262 ms. Trace:
I20260924 19:57:38.901132 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 24f84032bca24a7ca9da65f0bbceb0d5 (table default.Table_5f33677cfb6a4eab8497cae637ecdf26 [id=6e11b13ce19a4d12893da7865efea4b4]) successfully deleted
I20260924 19:57:38.936367 18776 rpcz_store.cc:276] 0924 19:57:37.638583 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.737808 (+ 99225us) service_pool.cc:224] Handling call
0924 19:57:38.900610 (+1162802us) ts_tablet_manager.cc:1151] Deleting tablet 5f994e0401df4fc19bc5abb10f08bc6e
0924 19:57:38.900657 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.900843 (+ 186us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161482}
W20260924 19:57:38.938282 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 5f994e0401df4fc19bc5abb10f08bc6e because the tablet was not found. No further retry: Not found: Tablet not found: 5f994e0401df4fc19bc5abb10f08bc6e
W20260924 19:57:38.940143 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 57008266b80345e0a3f8e60bd7e2d62b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57008266b80345e0a3f8e60bd7e2d62b already in progress: creating tablet
I20260924 19:57:38.941646 18776 tablet_replica.cc:333] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:38.951550 18746 raft_consensus.cc:493] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:38.952100 18746 raft_consensus.cc:515] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:38.954487 18746 leader_election.cc:290] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:38.957211 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edc6c16578e5463a9d44c25fd6c538a9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:38.959061 18050 leader_election.cc:343] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:39.000042 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0feecd2fdb4409f86f6a5fbfbab3537" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:39.002394 17753 leader_election.cc:343] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:39.004951 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0feecd2fdb4409f86f6a5fbfbab3537" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:39.005776 17970 raft_consensus.cc:2468] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:39.007875 17753 leader_election.cc:304] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters: eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:39.009397 18780 raft_consensus.cc:2804] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:39.009799 18780 raft_consensus.cc:493] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:39.010140 18780 raft_consensus.cc:3060] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.017941 18780 raft_consensus.cc:515] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.021102 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0feecd2fdb4409f86f6a5fbfbab3537" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
W20260924 19:57:39.022667 17753 leader_election.cc:343] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:38.966439 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edc6c16578e5463a9d44c25fd6c538a9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:39.024853 17818 raft_consensus.cc:2468] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:39.026403 18051 leader_election.cc:304] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:39.028095 18746 raft_consensus.cc:2804] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:39.028553 18746 raft_consensus.cc:493] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:39.028997 18746 raft_consensus.cc:3060] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.040928 18224 raft_consensus.cc:359] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.044178 18224 raft_consensus.cc:385] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:39.044536 18224 raft_consensus.cc:740] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:39.051961 18823 consensus_queue.cc:1048] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:39.054392 18220 ts_tablet_manager.cc:1434] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.386s user 0.008s sys 0.000s
I20260924 19:57:39.074527 18776 raft_consensus.cc:2243] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:39.075064 18776 raft_consensus.cc:2272] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:39.075687 18223 tablet_bootstrap.cc:492] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:39.076099 18223 ts_tablet_manager.cc:1403] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.244s user 0.012s sys 0.081s
I20260924 19:57:39.077019 18776 ts_tablet_manager.cc:1916] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:39.078254 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0feecd2fdb4409f86f6a5fbfbab3537" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:39.078815 17970 raft_consensus.cc:3060] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.078485 18223 raft_consensus.cc:359] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.079260 18223 raft_consensus.cc:385] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:39.079514 18223 raft_consensus.cc:740] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:39.082443 18223 consensus_queue.cc:260] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.084445 18223 ts_tablet_manager.cc:1434] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260924 19:57:39.097085 18223 tablet_bootstrap.cc:492] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:39.115399 18223 tablet_bootstrap.cc:654] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:39.158095 18776 ts_tablet_manager.cc:1929] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:39.165200 18220 tablet_bootstrap.cc:492] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:39.175207 18776 log.cc:1199] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/2cf62a73c49245198d65011875148dc3
I20260924 19:57:39.176558 18776 ts_tablet_manager.cc:1950] T 2cf62a73c49245198d65011875148dc3 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:39.087024 17970 raft_consensus.cc:2468] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:39.183343 18224 consensus_queue.cc:260] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.053006 18839 leader_election.cc:290] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:39.073653 18858 raft_consensus.cc:493] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:39.184442 18776 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 165) took 1069 ms. Trace:
I20260924 19:57:39.076871 18780 leader_election.cc:290] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:39.181527 17753 leader_election.cc:304] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters: eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:39.184798 18858 raft_consensus.cc:515] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.184767 18776 rpcz_store.cc:276] 0924 19:57:38.115086 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:38.124927 (+ 9841us) service_pool.cc:224] Handling call
0924 19:57:38.941454 (+816527us) ts_tablet_manager.cc:1151] Deleting tablet 2cf62a73c49245198d65011875148dc3
0924 19:57:38.941510 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:39.088784 (+147274us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:39.157894 (+ 69110us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:39.184367 (+ 26473us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":131819,"tablet-delete.queue_time_us":801799}
I20260924 19:57:38.737591 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbb82914df65416aaf47db8e507afb40" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:39.185945 17816 raft_consensus.cc:2468] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:39.186728 18858 leader_election.cc:290] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:39.205912 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07f86f24d9d14d05898e1c0f69428850" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:39.206424 18761 tablet.cc:1751] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:39.208240 18778 raft_consensus.cc:493] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:39.208729 18778 raft_consensus.cc:515] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.186771 18780 raft_consensus.cc:2804] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:39.213209 18780 raft_consensus.cc:697] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:39.213912 18778 leader_election.cc:290] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:39.214097 18780 consensus_queue.cc:237] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:39.219326 17753 leader_election.cc:343] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:39.232367 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07f86f24d9d14d05898e1c0f69428850" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:39.256098 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a765abd702644f079bddffb3fc4238ef" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:39.256763 18113 raft_consensus.cc:2468] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
W20260924 19:57:39.262140 17753 leader_election.cc:343] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:39.262589 17905 leader_election.cc:304] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:39.262645 17753 leader_election.cc:304] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:39.263648 18858 raft_consensus.cc:2749] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:39.263799 18785 raft_consensus.cc:2804] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:39.264133 18785 raft_consensus.cc:493] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:39.264459 18785 raft_consensus.cc:3060] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:39.271389 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 47140.8us
I20260924 19:57:39.273424 18051 leader_election.cc:304] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:39.273931 18823 raft_consensus.cc:493] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:39.275764 18785 raft_consensus.cc:515] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.279294 18823 raft_consensus.cc:515] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.281397 18867 raft_consensus.cc:2804] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:39.281450 18823 leader_election.cc:290] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:39.281761 18867 raft_consensus.cc:493] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:39.282174 18867 raft_consensus.cc:3060] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.038642 18746 raft_consensus.cc:515] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.286316 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edc6c16578e5463a9d44c25fd6c538a9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
W20260924 19:57:39.287508 18050 leader_election.cc:343] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:39.289022 18777 ts_tablet_manager.cc:1929] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:39.289463 18777 log.cc:1199] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:39.290266 18777 ts_tablet_manager.cc:1950] T 24f84032bca24a7ca9da65f0bbceb0d5 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:39.292047 18785 leader_election.cc:290] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:39.292440 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 153) took 1711 ms. Trace:
I20260924 19:57:39.294669 18777 rpcz_store.cc:276] 0924 19:57:37.580939 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.583005 (+ 2066us) service_pool.cc:224] Handling call
0924 19:57:38.850817 (+1267812us) ts_tablet_manager.cc:1151] Deleting tablet 24f84032bca24a7ca9da65f0bbceb0d5
0924 19:57:38.850855 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:39.269023 (+418168us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:39.288895 (+ 19872us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:39.292382 (+ 3487us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4472,"tablet-delete.queue_time_us":1267261}
I20260924 19:57:39.297192 18231 ts_tablet_manager.cc:1929] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:39.297575 18231 log.cc:1199] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/d970cc92a1134a98a613d198a319cb01
I20260924 19:57:39.298026 18777 tablet_replica.cc:333] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:39.300400 18224 ts_tablet_manager.cc:1434] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.521s user 0.008s sys 0.000s
I20260924 19:57:39.300367 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a765abd702644f079bddffb3fc4238ef" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
W20260924 19:57:39.300770 18224 ts_tablet_manager.cc:1467] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202: Tablet startup took 521ms
I20260924 19:57:39.301050 18113 raft_consensus.cc:3060] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.301529 18777 raft_consensus.cc:2243] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:39.303905 18871 tablet.cc:1751] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:39.309967 18113 raft_consensus.cc:2468] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:39.312016 18777 raft_consensus.cc:2272] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:39.314095 18777 ts_tablet_manager.cc:1916] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:39.318405 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "077f70bcee7e412299da12ddc30a63ff" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:39.319198 17816 raft_consensus.cc:2468] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:39.319362 18868 raft_consensus.cc:493] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:39.319787 18868 raft_consensus.cc:515] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.321648 18051 leader_election.cc:304] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:39.322865 18874 raft_consensus.cc:2804] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:39.323341 18874 raft_consensus.cc:493] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:39.323745 18874 raft_consensus.cc:3060] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.323853 18868 leader_election.cc:290] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:39.186239 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 2cf62a73c49245198d65011875148dc3 (table default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265]) successfully deleted
W20260924 19:57:39.327666 17599 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 141447us
I20260924 19:57:39.331089 18746 leader_election.cc:290] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:39.336982 18872 log.cc:927] Time spent T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
W20260924 19:57:39.357039 17906 leader_election.cc:343] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: a765abd702644f079bddffb3fc4238ef
I20260924 19:57:39.364625 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd2f4eb8bf254c4e85b6a9072271766b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:39.365343 17815 raft_consensus.cc:2468] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:39.377472 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 24f84032bca24a7ca9da65f0bbceb0d5 (table default.Table_5f33677cfb6a4eab8497cae637ecdf26 [id=6e11b13ce19a4d12893da7865efea4b4]) successfully deleted
W20260924 19:57:35.968530 18649 debug-util.cc:398] Leaking SignalData structure 0x7b08000379a0 after lost signal to thread 18320
I20260924 19:57:39.296169 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "077f70bcee7e412299da12ddc30a63ff" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:39.379099 18320 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 4.837s user 0.001s sys 0.000s
W20260924 19:57:39.380467 18320 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 4.839s user 0.001s sys 0.000s
I20260924 19:57:39.350433 18876 tablet.cc:1751] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:39.352785 18777 ts_tablet_manager.cc:1929] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:39.382014 18777 log.cc:1199] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/151247d6b5254cd884c58db9311ac04a
I20260924 19:57:39.382776 18777 ts_tablet_manager.cc:1950] T 151247d6b5254cd884c58db9311ac04a P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:39.383678 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 152) took 1803 ms. Trace:
I20260924 19:57:39.384016 18777 rpcz_store.cc:276] 0924 19:57:37.580697 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.583647 (+ 2950us) service_pool.cc:224] Handling call
0924 19:57:39.297864 (+1714217us) ts_tablet_manager.cc:1151] Deleting tablet 151247d6b5254cd884c58db9311ac04a
0924 19:57:39.297912 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:39.336091 (+ 38179us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:39.352671 (+ 16580us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:39.383611 (+ 30940us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9757,"tablet-delete.queue_time_us":1713656}
I20260924 19:57:39.385577 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 151) took 1805 ms. Trace:
I20260924 19:57:39.385753 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 151247d6b5254cd884c58db9311ac04a (table default.Table_c8e27940040c495780f92f92431541cb [id=47a10be86ad441a7961c0f34fed46040]) successfully deleted
I20260924 19:57:39.385914 18777 rpcz_store.cc:276] 0924 19:57:37.580444 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.584278 (+ 3834us) service_pool.cc:224] Handling call
0924 19:57:39.385268 (+1800990us) ts_tablet_manager.cc:1151] Deleting tablet a2e4519aed7642678db3a518a89778de
0924 19:57:39.385316 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:39.385519 (+ 203us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1800475}
I20260924 19:57:39.356649 18867 raft_consensus.cc:515] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.361801 18874 raft_consensus.cc:515] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.388728 18867 leader_election.cc:290] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:39.389914 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 158) took 1807 ms. Trace:
I20260924 19:57:39.390098 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edc6c16578e5463a9d44c25fd6c538a9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:39.390674 17815 raft_consensus.cc:3060] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.390259 18777 rpcz_store.cc:276] 0924 19:57:37.582139 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.604333 (+ 22194us) service_pool.cc:224] Handling call
0924 19:57:39.389615 (+1785282us) ts_tablet_manager.cc:1151] Deleting tablet ef3f82de70bb4328a15a044fe0e500bf
0924 19:57:39.389658 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:39.389850 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1784309}
I20260924 19:57:39.394906 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 159) took 1788 ms. Trace:
W20260924 19:57:39.301028 18224 ts_tablet_manager.cc:1469] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202: Trace:
0924 19:57:34.825905 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:34.826057 (+ 152us) service_pool.cc:224] Handling call
0924 19:57:35.006096 (+180039us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:35.006199 (+ 103us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:35.108751 (+102552us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:35.218864 (+110113us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:35.219330 (+ 466us) inbound_call.cc:177] Queueing success response
0924 19:57:38.428627 (+3209297us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0924 19:57:38.778975 (+350348us) ts_tablet_manager.cc:1435] Starting tablet replica
0924 19:57:38.779626 (+ 651us) tablet_replica.cc:261] Starting instrumentation
0924 19:57:39.039645 (+260019us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":1323,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":224,"tablet-open.queue_time_us":3209386}
I20260924 19:57:39.395270 18777 rpcz_store.cc:276] 0924 19:57:37.606200 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.606339 (+ 139us) service_pool.cc:224] Handling call
0924 19:57:39.394321 (+1787982us) ts_tablet_manager.cc:1151] Deleting tablet efd7339d0fee436299a7eac02a82addd
0924 19:57:39.394676 (+ 355us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:39.394851 (+ 175us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1787162}
I20260924 19:57:39.398437 18224 tablet_bootstrap.cc:492] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:34.423591 18653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:57:39.399247 18874 leader_election.cc:290] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:39.400694 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 150) took 1820 ms. Trace:
I20260924 19:57:39.401088 18777 rpcz_store.cc:276] 0924 19:57:37.580158 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.623182 (+ 43024us) service_pool.cc:224] Handling call
0924 19:57:39.398354 (+1775172us) ts_tablet_manager.cc:1151] Deleting tablet f482b53a4a2d44f6b3431c588cd605ce
0924 19:57:39.400440 (+ 2086us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:39.400631 (+ 191us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773934}
I20260924 19:57:39.401959 18220 tablet_bootstrap.cc:654] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:39.403349 18231 ts_tablet_manager.cc:1950] T d970cc92a1134a98a613d198a319cb01 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:39.405674 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 159) took 1886 ms. Trace:
I20260924 19:57:39.406845 18777 tablet_replica.cc:333] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:39.408406 18777 raft_consensus.cc:2243] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:39.410108 18777 pending_rounds.cc:70] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: Trying to abort 1 pending ops.
I20260924 19:57:39.412145 18051 leader_election.cc:304] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:39.412180 18777 pending_rounds.cc:77] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986299379363840 op_type: NO_OP noop_request { }
I20260924 19:57:39.413460 18777 raft_consensus.cc:2889] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:39.413861 18777 raft_consensus.cc:2272] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:39.416033 18777 ts_tablet_manager.cc:1916] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:39.312657 17905 leader_election.cc:304] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
W20260924 19:57:39.418648 17906 leader_election.cc:343] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: a765abd702644f079bddffb3fc4238ef
W20260924 19:57:39.420308 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 60340.6us
W20260924 19:57:39.420658 18877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:57:39.425458 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 112794us
I20260924 19:57:39.427145 18778 raft_consensus.cc:2804] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:39.430508 18778 raft_consensus.cc:697] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:39.431561 18778 consensus_queue.cc:237] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.433940 18320 server_base.cc:1061] running on GCE node
I20260924 19:57:39.435716 18320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:57:39.445839 18220 tablet_bootstrap.cc:492] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
W20260924 19:57:39.424000 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet a2e4519aed7642678db3a518a89778de because the tablet was not found. No further retry: Not found: Tablet not found: a2e4519aed7642678db3a518a89778de
I20260924 19:57:39.446538 18220 ts_tablet_manager.cc:1403] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.282s user 0.012s sys 0.001s
W20260924 19:57:39.446900 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet efd7339d0fee436299a7eac02a82addd because the tablet was not found. No further retry: Not found: Tablet not found: efd7339d0fee436299a7eac02a82addd
I20260924 19:57:39.447229 18320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260924 19:57:39.447821 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet f482b53a4a2d44f6b3431c588cd605ce because the tablet was not found. No further retry: Not found: Tablet not found: f482b53a4a2d44f6b3431c588cd605ce
I20260924 19:57:39.417605 18867 raft_consensus.cc:2804] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:39.454396 18867 raft_consensus.cc:493] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:39.454751 18867 raft_consensus.cc:3060] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.454895 18320 hybrid_clock.cc:648] HybridClock initialized: now 1790279859454786 us; error 93 us; skew 500 ppm
I20260924 19:57:39.456087 18320 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:57:39.425958 18223 tablet_bootstrap.cc:492] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:39.460527 18223 ts_tablet_manager.cc:1403] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.365s user 0.022s sys 0.016s
W20260924 19:57:39.426723 18050 leader_election.cc:343] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:39.463759 18867 raft_consensus.cc:515] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.464568 18220 raft_consensus.cc:359] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.465243 18220 raft_consensus.cc:385] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:39.465547 18220 raft_consensus.cc:740] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:39.466616 18867 leader_election.cc:290] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:39.466320 18220 consensus_queue.cc:260] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.469290 18220 ts_tablet_manager.cc:1434] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260924 19:57:39.476956 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbb82914df65416aaf47db8e507afb40" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:39.478583 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd2f4eb8bf254c4e85b6a9072271766b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:39.479967 18220 tablet_bootstrap.cc:492] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:39.486775 18220 tablet_bootstrap.cc:654] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:39.490479 18050 consensus_peers.cc:597] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:39.469098 18223 raft_consensus.cc:359] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.490899 18223 raft_consensus.cc:385] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:39.491189 18223 raft_consensus.cc:740] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:39.491806 18223 consensus_queue.cc:260] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:39.493152 18050 leader_election.cc:343] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:39.493844 18223 ts_tablet_manager.cc:1434] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
W20260924 19:57:39.494560 18050 leader_election.cc:343] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:39.494763 18223 tablet_bootstrap.cc:492] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:39.409940 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbb82914df65416aaf47db8e507afb40" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:39.505182 17818 raft_consensus.cc:3060] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.411253 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "077f70bcee7e412299da12ddc30a63ff" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:39.509060 17816 raft_consensus.cc:3060] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.516830 17816 raft_consensus.cc:2468] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:39.517343 18223 tablet_bootstrap.cc:654] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:39.487335 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "077f70bcee7e412299da12ddc30a63ff" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:39.492224 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd2f4eb8bf254c4e85b6a9072271766b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
W20260924 19:57:39.524956 18050 leader_election.cc:343] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:39.502866 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd2f4eb8bf254c4e85b6a9072271766b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
W20260924 19:57:39.525964 18050 leader_election.cc:343] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:39.526019 17821 raft_consensus.cc:3060] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.533344 17821 raft_consensus.cc:2468] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:39.537124 18051 leader_election.cc:304] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:39.537835 18874 raft_consensus.cc:2804] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:39.523782 17818 raft_consensus.cc:2468] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:39.538259 18874 raft_consensus.cc:697] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:39.524983 18231 rpcz_store.cc:276] 0924 19:57:37.519256 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.519363 (+ 107us) service_pool.cc:224] Handling call
0924 19:57:38.385377 (+866014us) ts_tablet_manager.cc:1151] Deleting tablet d970cc92a1134a98a613d198a319cb01
0924 19:57:38.385421 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:38.680678 (+295257us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:39.297067 (+616389us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:39.405613 (+108546us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71482,"tablet-delete.queue_time_us":865464}
I20260924 19:57:39.539391 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet d970cc92a1134a98a613d198a319cb01 (table default.Table_a50429edeaf645668bc84244412bb359 [id=1853f611de514a55afd7a68d742503b8]) successfully deleted
I20260924 19:57:39.539058 18874 consensus_queue.cc:237] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.540242 18231 tablet_replica.cc:333] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:39.541021 18231 raft_consensus.cc:2243] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:39.543119 18320 webserver.cc:533] Webserver started at http://127.17.2.190:37347/ using document root <none> and password file <none>
I20260924 19:57:39.544934 18320 fs_manager.cc:362] Metadata directory not provided
I20260924 19:57:39.545292 18320 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:57:39.558679 17949 tablet_service.cc:1511] Processing CreateTablet for tablet 89646893588340038f30ed4754623ec2 (DEFAULT_TABLE table=default.Table_bb9775fce761443ca3a306e0cace741e [id=dc2b2e2afb22446ba04263f4f86a23e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:39.559705 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89646893588340038f30ed4754623ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:39.559849 18051 leader_election.cc:304] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:39.561024 18746 raft_consensus.cc:2804] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:39.562806 17615 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:39.564594 18746 raft_consensus.cc:697] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:39.565423 18746 consensus_queue.cc:237] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.565889 17615 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:39.570420 18051 leader_election.cc:304] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:39.571347 18874 raft_consensus.cc:2804] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:39.490320 17815 raft_consensus.cc:2468] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:39.572366 18231 raft_consensus.cc:2272] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:39.575112 18231 ts_tablet_manager.cc:1916] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:39.579931 18874 raft_consensus.cc:697] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:39.580761 18874 consensus_queue.cc:237] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.585263 17966 raft_consensus.cc:3060] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:39.587661 18224 tablet_bootstrap.cc:654] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:39.600268 18051 leader_election.cc:304] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:39.601588 18873 raft_consensus.cc:2804] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:39.602108 18873 raft_consensus.cc:697] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:39.602941 18873 consensus_queue.cc:237] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.610183 18320 fs_manager.cc:714] Time spent opening directory manager: real 0.057s user 0.012s sys 0.006s
I20260924 19:57:39.626199 18220 tablet_bootstrap.cc:492] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:39.626273 18223 tablet_bootstrap.cc:492] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:39.626705 18220 ts_tablet_manager.cc:1403] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.147s user 0.017s sys 0.006s
I20260924 19:57:39.626746 18223 ts_tablet_manager.cc:1403] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.132s user 0.008s sys 0.003s
I20260924 19:57:39.629159 18220 raft_consensus.cc:359] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.634922 17612 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:39.637858 17612 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:39.639300 18220 raft_consensus.cc:385] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:39.639276 17612 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet c1684aaf9d984649bff8c4b0b12044b9 (table default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe]). Expected version 1 got 0
I20260924 19:57:39.639667 18220 raft_consensus.cc:740] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:39.640486 17612 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
I20260924 19:57:39.640484 18220 consensus_queue.cc:260] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.642146 18220 ts_tablet_manager.cc:1434] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20260924 19:57:39.644979 18220 tablet_bootstrap.cc:492] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:39.645110 18894 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:57:39.648521 18320 fs_manager.cc:730] Time spent opening block manager: real 0.035s user 0.002s sys 0.003s
I20260924 19:57:39.649058 18320 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal
uuid: "3e059bbe4dac4178bccb3be48cb1118d"
format_stamp: "Formatted at 2026-09-24 19:57:09 on dist-test-slave-h376"
I20260924 19:57:39.652197 18320 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:57:39.670804 18220 tablet_bootstrap.cc:654] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:39.677575 18084 tablet_service.cc:1511] Processing CreateTablet for tablet 89646893588340038f30ed4754623ec2 (DEFAULT_TABLE table=default.Table_bb9775fce761443ca3a306e0cace741e [id=dc2b2e2afb22446ba04263f4f86a23e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:39.678855 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89646893588340038f30ed4754623ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:39.664312 18223 raft_consensus.cc:359] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.682716 18223 raft_consensus.cc:385] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:57:39.680029 17602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260924 19:57:39.682998 18223 raft_consensus.cc:740] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:39.693984 18223 consensus_queue.cc:260] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.698524 18220 tablet_bootstrap.cc:492] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:39.698490 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 89646893588340038f30ed4754623ec2 (DEFAULT_TABLE table=default.Table_bb9775fce761443ca3a306e0cace741e [id=dc2b2e2afb22446ba04263f4f86a23e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:39.699082 18220 ts_tablet_manager.cc:1403] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.054s user 0.011s sys 0.000s
I20260924 19:57:39.699622 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89646893588340038f30ed4754623ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:39.702699 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:39.708884 18223 ts_tablet_manager.cc:1434] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.082s user 0.005s sys 0.002s
I20260924 19:57:39.712188 18223 tablet_bootstrap.cc:492] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:39.714368 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:39.719767 18220 raft_consensus.cc:359] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.720504 18220 raft_consensus.cc:385] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:39.720774 18220 raft_consensus.cc:740] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:39.721493 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:39.721463 18220 consensus_queue.cc:260] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.722506 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:39.728603 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet c1684aaf9d984649bff8c4b0b12044b9 (table default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe]). Expected version 1 got 0
I20260924 19:57:39.732281 18220 ts_tablet_manager.cc:1434] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260924 19:57:39.733237 18220 tablet_bootstrap.cc:492] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:39.737015 17611 catalog_manager.cc:5838] T 3b328d37d55e4116a29322caaa21d431 P c4f6e3a7b06148918d230c77ab0c8202 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:39.738910 18220 tablet_bootstrap.cc:654] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:39.739946 18224 tablet_bootstrap.cc:492] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:39.740386 18224 ts_tablet_manager.cc:1403] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.343s user 0.010s sys 0.000s
I20260924 19:57:39.742327 18223 tablet_bootstrap.cc:654] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:39.743544 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
I20260924 19:57:39.755854 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:39.758399 18086 tablet_service.cc:1511] Processing CreateTablet for tablet a9e7c5b28fea4dbda802702c8b85ec54 (DEFAULT_TABLE table=default.Table_5208198cb618443dbec0f760012e4edd [id=7cb855f3577d4824ac7edb72b8f7072c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:39.759676 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e7c5b28fea4dbda802702c8b85ec54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:39.764047 17614 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 200) took 1574 ms. Trace:
I20260924 19:57:39.764374 17614 rpcz_store.cc:276] 0924 19:57:38.189322 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:38.224389 (+ 35067us) service_pool.cc:224] Handling call
0924 19:57:38.391414 (+167025us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa5522077e6 00007fa54fdbc634 00007fa54fdbc5c2 00007fa54fdbc4f1 00007fa54fdbc46c 00007fa54fdbb722 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:38.392181 (+ 767us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:38.393289 (+ 1108us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:38.403783 (+ 10494us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:38.404003 (+ 220us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:38.466067 (+ 62064us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:39.763984 (+1297917us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:38.427274 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:38.433348 (+ 6074us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:38.433376 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:57:38.433389 (+ 13us) tablet.cc:662] Decoding operations
0924 19:57:38.433680 (+ 291us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:38.433725 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:38.433749 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:38.433938 (+ 189us) tablet.cc:701] Row locks acquired
0924 19:57:38.433953 (+ 15us) write_op.cc:260] PREPARE: finished
0924 19:57:38.434088 (+ 135us) write_op.cc:270] Start()
0924 19:57:38.434182 (+ 94us) write_op.cc:276] Timestamp: P: 1790279858434026 usec, L: 0
0924 19:57:38.434206 (+ 24us) op_driver.cc:348] REPLICATION: starting
0924 19:57:38.434995 (+ 789us) log.cc:844] Serialized 405 byte log entry
0924 19:57:38.462467 (+ 27472us) op_driver.cc:464] REPLICATION: finished
0924 19:57:38.463568 (+ 1101us) write_op.cc:301] APPLY: starting
0924 19:57:38.463618 (+ 50us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:38.463710 (+ 92us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:38.463811 (+ 101us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:38.463940 (+ 129us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:38.464055 (+ 115us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:38.464101 (+ 46us) write_op.cc:312] APPLY: finished
0924 19:57:38.464242 (+ 141us) log.cc:844] Serialized 23 byte log entry
0924 19:57:38.464377 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.464430 (+ 53us) write_op.cc:454] Released schema lock
0924 19:57:38.464461 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:38.503683 (+ 39222us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.503713 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":379429632,"child_traces":[["op",{"apply.queue_time_us":965,"apply.run_cpu_time_us":1067,"apply.run_wall_time_us":1068,"mutex_wait_us":66518,"num_ops":1,"prepare.queue_time_us":21626,"prepare.run_cpu_time_us":2465,"prepare.run_wall_time_us":76375,"raft.queue_time_us":66790,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":13,"replication_time_us":28185}]]}
I20260924 19:57:39.775476 17607 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet c884f12860164f12bbcd0d2f87067013
I20260924 19:57:39.777004 17607 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 204) took 1182 ms. Trace:
I20260924 19:57:39.782732 17796 tablet_service.cc:1558] Processing DeleteTablet for tablet c884f12860164f12bbcd0d2f87067013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:38 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:39.785928 18082 tablet_service.cc:1558] Processing DeleteTablet for tablet c884f12860164f12bbcd0d2f87067013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:38 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:39.777307 17607 rpcz_store.cc:276] 0924 19:57:38.594262 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:38.625015 (+ 30753us) service_pool.cc:224] Handling call
0924 19:57:38.633392 (+ 8377us) spinlock_profiling.cc:243] Waited 7.57 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552207476 00007fa54fdb7b34 00007fa54fdb7ac2 00007fa54fdb79f1 00007fa54fdb796c 00007fa54fdb6c22 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:38.635882 (+ 2490us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:38.636410 (+ 528us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:38.636547 (+ 137us) catalog_manager.cc:2863] Locking tablets
0924 19:57:39.098205 (+461658us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:39.164014 (+ 65809us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:39.164282 (+ 268us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:39.775371 (+611089us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:39.776920 (+ 1549us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:39.111735 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:39.112217 (+ 482us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:39.112245 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:57:39.112259 (+ 14us) tablet.cc:662] Decoding operations
0924 19:57:39.112645 (+ 386us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:39.112687 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:39.112717 (+ 30us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:39.113397 (+ 680us) tablet.cc:701] Row locks acquired
0924 19:57:39.113415 (+ 18us) write_op.cc:260] PREPARE: finished
0924 19:57:39.113548 (+ 133us) write_op.cc:270] Start()
0924 19:57:39.113637 (+ 89us) write_op.cc:276] Timestamp: P: 1790279859113470 usec, L: 0
0924 19:57:39.113657 (+ 20us) op_driver.cc:348] REPLICATION: starting
0924 19:57:39.114209 (+ 552us) log.cc:844] Serialized 768 byte log entry
0924 19:57:39.160040 (+ 45831us) op_driver.cc:464] REPLICATION: finished
0924 19:57:39.161617 (+ 1577us) write_op.cc:301] APPLY: starting
0924 19:57:39.161698 (+ 81us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:39.161840 (+ 142us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:39.161950 (+ 110us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:39.162260 (+ 310us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:39.162431 (+ 171us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:39.162472 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:57:39.162712 (+ 240us) log.cc:844] Serialized 29 byte log entry
0924 19:57:39.162884 (+ 172us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:39.162967 (+ 83us) write_op.cc:454] Released schema lock
0924 19:57:39.163005 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:39.166906 (+ 3901us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:39.166945 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":17411328,"child_traces":[["op",{"apply.queue_time_us":1388,"apply.run_cpu_time_us":1690,"apply.run_wall_time_us":2840,"num_ops":2,"prepare.queue_time_us":11444,"prepare.run_cpu_time_us":3324,"prepare.run_wall_time_us":55139,"raft.queue_time_us":48394,"raft.run_cpu_time_us":876,"raft.run_wall_time_us":877,"replication_time_us":46265,"thread_start_us":19494,"threads_started":1}]]}
I20260924 19:57:39.795398 17796 tablet_service.cc:1511] Processing CreateTablet for tablet a9e7c5b28fea4dbda802702c8b85ec54 (DEFAULT_TABLE table=default.Table_5208198cb618443dbec0f760012e4edd [id=7cb855f3577d4824ac7edb72b8f7072c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:39.796465 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e7c5b28fea4dbda802702c8b85ec54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:39.798580 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 78801b8d0b664217a74b20160ba2d8ac (DEFAULT_TABLE table=default.Table_6c2fccb54ef24d088cd2ec174a8e20e9 [id=81b26242949442c0a7db6f55f0f18097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:39.799718 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78801b8d0b664217a74b20160ba2d8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:39.806878 18873 raft_consensus.cc:493] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:39.807538 18873 raft_consensus.cc:515] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.810753 18873 leader_election.cc:290] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:39.811388 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0feecd2fdb4409f86f6a5fbfbab3537" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:39.831450 18224 raft_consensus.cc:359] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.832139 18224 raft_consensus.cc:385] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:39.832401 18224 raft_consensus.cc:740] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:39.833115 18224 consensus_queue.cc:260] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:39.835935 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f994e0401df4fc19bc5abb10f08bc6e with delete_type TABLET_DATA_DELETED (replaced by tablet e1137318a9fa4a0781f718dc34c4ecbe) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:39.849325 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:39.855096 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet 349c6e9413bb41268d0c41d549203e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:39.857100 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:39.860453 17610 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 8a524bad1c4040e58ad84fcf5a4fc47d
I20260924 19:57:39.861959 17610 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 199) took 2400 ms. Trace:
I20260924 19:57:39.862257 17610 rpcz_store.cc:276] 0924 19:57:37.461121 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.994358 (+533237us) service_pool.cc:224] Handling call
0924 19:57:37.995860 (+ 1502us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:37.996588 (+ 728us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:37.996727 (+ 139us) catalog_manager.cc:2863] Locking tablets
0924 19:57:38.392327 (+395600us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:38.465074 (+ 72747us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:38.465306 (+ 232us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:39.860341 (+1395035us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:39.861901 (+ 1560us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:38.403281 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:38.403744 (+ 463us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:38.403774 (+ 30us) write_op.cc:435] Acquired schema lock
0924 19:57:38.403787 (+ 13us) tablet.cc:662] Decoding operations
0924 19:57:38.404165 (+ 378us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:38.404209 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:38.404237 (+ 28us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:38.420520 (+ 16283us) tablet.cc:701] Row locks acquired
0924 19:57:38.420539 (+ 19us) write_op.cc:260] PREPARE: finished
0924 19:57:38.420677 (+ 138us) write_op.cc:270] Start()
0924 19:57:38.420776 (+ 99us) write_op.cc:276] Timestamp: P: 1790279858420621 usec, L: 0
0924 19:57:38.420797 (+ 21us) op_driver.cc:348] REPLICATION: starting
0924 19:57:38.423669 (+ 2872us) log.cc:844] Serialized 793 byte log entry
0924 19:57:38.462222 (+ 38553us) op_driver.cc:464] REPLICATION: finished
0924 19:57:38.462674 (+ 452us) write_op.cc:301] APPLY: starting
0924 19:57:38.462729 (+ 55us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:38.462850 (+ 121us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:38.462967 (+ 117us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:38.463253 (+ 286us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:38.463386 (+ 133us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:38.463434 (+ 48us) write_op.cc:312] APPLY: finished
0924 19:57:38.463604 (+ 170us) log.cc:844] Serialized 29 byte log entry
0924 19:57:38.463735 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.463801 (+ 66us) write_op.cc:454] Released schema lock
0924 19:57:38.463831 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:38.464055 (+ 224us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.464079 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":309,"apply.run_cpu_time_us":1347,"apply.run_wall_time_us":1347,"num_ops":2,"prepare.queue_time_us":9006,"prepare.run_cpu_time_us":2601,"prepare.run_wall_time_us":23831,"raft.queue_time_us":15333,"raft.run_cpu_time_us":992,"raft.run_wall_time_us":87742,"replication_time_us":41322}]]}
I20260924 19:57:39.867383 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a524bad1c4040e58ad84fcf5a4fc47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:37 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:39.868811 17610 master_service.cc:960] ReplaceTablet: received request to replace tablet dd2f4eb8bf254c4e85b6a9072271766b from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:39.869956 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a524bad1c4040e58ad84fcf5a4fc47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:37 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:39.872622 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet d970cc92a1134a98a613d198a319cb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:39.896499 17801 tablet_service.cc:1511] Processing CreateTablet for tablet 78801b8d0b664217a74b20160ba2d8ac (DEFAULT_TABLE table=default.Table_6c2fccb54ef24d088cd2ec174a8e20e9 [id=81b26242949442c0a7db6f55f0f18097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:39.897974 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78801b8d0b664217a74b20160ba2d8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:39.899087 18746 raft_consensus.cc:493] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:39.899585 18746 raft_consensus.cc:515] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:39.901503 18746 leader_election.cc:290] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:39.916965 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd6e656a653a43d1843a91fee813237d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
W20260924 19:57:39.874900 17613 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 39) took 3063 ms (client timeout 1000 ms). Trace:
I20260924 19:57:39.925531 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:39.927186 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet 349c6e9413bb41268d0c41d549203e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:39.929216 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet 151247d6b5254cd884c58db9311ac04a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:39.928964 17608 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 37) took 2181 ms (client timeout 1000 ms). Trace:
I20260924 19:57:39.936134 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet d970cc92a1134a98a613d198a319cb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:39.936867 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet 41bdb846ca19425b882434df0167479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:31 UTC) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:39.932060 17608 rpcz_store.cc:269] 0924 19:57:37.747933 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.955186 (+207253us) service_pool.cc:224] Handling call
0924 19:57:38.001539 (+ 46353us) spinlock_profiling.cc:243] Waited 43 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:39.928892 (+1927353us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:38.699541 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:38.700047 (+ 506us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:38.700075 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:57:38.700094 (+ 19us) tablet.cc:662] Decoding operations
0924 19:57:38.700678 (+ 584us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:38.700724 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:38.700751 (+ 27us) tablet.cc:685] Acquiring locks for 3 operations
0924 19:57:38.701138 (+ 387us) tablet.cc:701] Row locks acquired
0924 19:57:38.701152 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:38.701290 (+ 138us) write_op.cc:270] Start()
0924 19:57:38.701404 (+ 114us) write_op.cc:276] Timestamp: P: 1790279858701200 usec, L: 0
0924 19:57:38.701420 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:38.702121 (+ 701us) log.cc:844] Serialized 1284 byte log entry
0924 19:57:39.090848 (+388727us) op_driver.cc:464] REPLICATION: finished
0924 19:57:39.091310 (+ 462us) write_op.cc:301] APPLY: starting
0924 19:57:39.091366 (+ 56us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:39.091745 (+ 379us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:39.091847 (+ 102us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:39.092244 (+ 397us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:39.092399 (+ 155us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0924 19:57:39.092440 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:57:39.092645 (+ 205us) log.cc:844] Serialized 35 byte log entry
0924 19:57:39.092826 (+ 181us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:39.092902 (+ 76us) write_op.cc:454] Released schema lock
0924 19:57:39.092934 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:39.093160 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:39.093184 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":40532,"reported_tablets":21,"spinlock_wait_cycles":99010944,"child_traces":[["op",{"apply.queue_time_us":303,"apply.run_cpu_time_us":1629,"apply.run_wall_time_us":1816,"num_ops":3,"prepare.queue_time_us":13232,"prepare.run_cpu_time_us":2916,"prepare.run_wall_time_us":187456,"raft.queue_time_us":129963,"raft.run_cpu_time_us":16,"raft.run_wall_time_us":17,"replication_time_us":389325,"spinlock_wait_cycles":1920}]]}
I20260924 19:57:39.940850 18223 tablet_bootstrap.cc:492] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:39.941256 18223 ts_tablet_manager.cc:1403] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.229s user 0.003s sys 0.006s
I20260924 19:57:39.964635 18777 ts_tablet_manager.cc:1929] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:39.965122 18777 log.cc:1199] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/2cf62a73c49245198d65011875148dc3
I20260924 19:57:39.979543 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 41bdb846ca19425b882434df0167479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:31 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:39.979542 17941 tablet_service.cc:1558] Processing DeleteTablet for tablet c884f12860164f12bbcd0d2f87067013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:38 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:39.982003 17942 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:39.983095 17943 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a524bad1c4040e58ad84fcf5a4fc47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:37 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:39.982870 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd6e656a653a43d1843a91fee813237d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:39.984148 17944 tablet_service.cc:1558] Processing DeleteTablet for tablet 349c6e9413bb41268d0c41d549203e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:39.987726 17946 tablet_service.cc:1558] Processing DeleteTablet for tablet a2e4519aed7642678db3a518a89778de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:29 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:39.986325 17945 tablet_service.cc:1511] Processing CreateTablet for tablet 78801b8d0b664217a74b20160ba2d8ac (DEFAULT_TABLE table=default.Table_6c2fccb54ef24d088cd2ec174a8e20e9 [id=81b26242949442c0a7db6f55f0f18097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:39.999717 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0feecd2fdb4409f86f6a5fbfbab3537" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:40.000670 17970 raft_consensus.cc:2393] T c0feecd2fdb4409f86f6a5fbfbab3537 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eaf64bfb64aa42948c9bc1411ce4ce35 in current term 1: Already voted for candidate c4f6e3a7b06148918d230c77ab0c8202 in this term.
I20260924 19:57:40.002013 18223 raft_consensus.cc:359] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.002715 18223 raft_consensus.cc:385] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:40.002967 18223 raft_consensus.cc:740] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:40.004299 17949 tablet_service.cc:1511] Processing CreateTablet for tablet a9e7c5b28fea4dbda802702c8b85ec54 (DEFAULT_TABLE table=default.Table_5208198cb618443dbec0f760012e4edd [id=7cb855f3577d4824ac7edb72b8f7072c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:40.013002 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78801b8d0b664217a74b20160ba2d8ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:40.033092 18224 ts_tablet_manager.cc:1434] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.292s user 0.007s sys 0.000s
I20260924 19:57:40.034381 18223 consensus_queue.cc:260] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.039855 18223 ts_tablet_manager.cc:1434] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.098s user 0.007s sys 0.000s
I20260924 19:57:39.966322 18777 ts_tablet_manager.cc:1950] T 2cf62a73c49245198d65011875148dc3 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:40.044850 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 166) took 1919 ms. Trace:
I20260924 19:57:40.045656 18320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:57:40.046185 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_05427967e4cc480cba4d8f7128b23677"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:40.050810 18780 raft_consensus.cc:493] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:40.051350 18780 raft_consensus.cc:515] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.052580 18320 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:57:40.053118 18320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:57:40.054344 18780 leader_election.cc:290] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:39.923606 17613 rpcz_store.cc:269] 0924 19:57:36.811601 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.259903 (+448302us) service_pool.cc:224] Handling call
0924 19:57:39.874764 (+2614861us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:37.594835 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:37.595346 (+ 511us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:37.595384 (+ 38us) write_op.cc:435] Acquired schema lock
0924 19:57:37.595401 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:37.595696 (+ 295us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:37.595741 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:37.595766 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:37.595931 (+ 165us) tablet.cc:701] Row locks acquired
0924 19:57:37.595943 (+ 12us) write_op.cc:260] PREPARE: finished
0924 19:57:37.596075 (+ 132us) write_op.cc:270] Start()
0924 19:57:37.596163 (+ 88us) write_op.cc:276] Timestamp: P: 1790279857595997 usec, L: 0
0924 19:57:37.596181 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:37.596701 (+ 520us) log.cc:844] Serialized 356 byte log entry
0924 19:57:37.841090 (+244389us) op_driver.cc:464] REPLICATION: finished
0924 19:57:37.914201 (+ 73111us) write_op.cc:301] APPLY: starting
0924 19:57:37.914742 (+ 541us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:37.914840 (+ 98us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:37.914965 (+ 125us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:37.915150 (+ 185us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:37.915345 (+ 195us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:37.915401 (+ 56us) write_op.cc:312] APPLY: finished
0924 19:57:37.915645 (+ 244us) log.cc:844] Serialized 23 byte log entry
0924 19:57:37.915862 (+ 217us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:37.915947 (+ 85us) write_op.cc:454] Released schema lock
0924 19:57:37.915993 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:37.981209 (+ 65216us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:37.981245 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":72903,"apply.run_cpu_time_us":2095,"apply.run_wall_time_us":66989,"mutex_wait_us":429,"num_ops":1,"prepare.queue_time_us":14678,"prepare.run_cpu_time_us":2097,"prepare.run_wall_time_us":2097,"replication_time_us":244782,"thread_start_us":38600,"threads_started":1}]]}
I20260924 19:57:39.981966 17606 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 198) took 2696 ms. Trace:
I20260924 19:57:40.059275 17613 master_service.cc:960] ReplaceTablet: received request to replace tablet 8a524bad1c4040e58ad84fcf5a4fc47d from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:40.060611 17613 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51280 (request call id 207) took 1465 ms. Trace:
I20260924 19:57:40.060899 17613 rpcz_store.cc:276] 0924 19:57:38.595071 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.059143 (+1464072us) service_pool.cc:224] Handling call
0924 19:57:40.060535 (+ 1392us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:40.058243 17606 rpcz_store.cc:276] 0924 19:57:37.286021 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.675460 (+389439us) service_pool.cc:224] Handling call
0924 19:57:37.754389 (+ 78929us) spinlock_profiling.cc:243] Waited 78 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa5522077e6 00007fa54fdbc634 00007fa54fdbc5c2 00007fa54fdbc4f1 00007fa54fdbc46c 00007fa54fdbb722 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:37.757008 (+ 2619us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:37.757731 (+ 723us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:37.758291 (+ 560us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:37.758482 (+ 191us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:38.312790 (+554308us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:39.981895 (+1669105us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:38.084746 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:38.085177 (+ 431us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:38.085203 (+ 26us) write_op.cc:435] Acquired schema lock
0924 19:57:38.085220 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:38.163407 (+ 78187us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:38.163450 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:38.163474 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:38.163657 (+ 183us) tablet.cc:701] Row locks acquired
0924 19:57:38.163680 (+ 23us) write_op.cc:260] PREPARE: finished
0924 19:57:38.163824 (+ 144us) write_op.cc:270] Start()
0924 19:57:38.163934 (+ 110us) write_op.cc:276] Timestamp: P: 1790279858163733 usec, L: 0
0924 19:57:38.163955 (+ 21us) op_driver.cc:348] REPLICATION: starting
0924 19:57:38.164525 (+ 570us) log.cc:844] Serialized 405 byte log entry
0924 19:57:38.309020 (+144495us) op_driver.cc:464] REPLICATION: finished
0924 19:57:38.311593 (+ 2573us) write_op.cc:301] APPLY: starting
0924 19:57:38.311637 (+ 44us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:38.311732 (+ 95us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:38.311825 (+ 93us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:38.311997 (+ 172us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:38.312136 (+ 139us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:38.312189 (+ 53us) write_op.cc:312] APPLY: finished
0924 19:57:38.312365 (+ 176us) log.cc:844] Serialized 23 byte log entry
0924 19:57:38.312509 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.312568 (+ 59us) write_op.cc:454] Released schema lock
0924 19:57:38.312602 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":179444992,"child_traces":[["op",{"apply.queue_time_us":2439,"apply.run_cpu_time_us":1188,"apply.run_wall_time_us":1188,"num_ops":1,"prepare.queue_time_us":207585,"prepare.run_cpu_time_us":2172,"prepare.run_wall_time_us":80016,"replication_time_us":144951}]]}
I20260924 19:57:40.064343 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_bdae18e3011a421d96cdcc107e8f1dc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:40.065995 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdae18e3011a421d96cdcc107e8f1dc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:40.070232 18224 tablet_bootstrap.cc:492] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:40.077554 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05427967e4cc480cba4d8f7128b23677 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:40.079116 17609 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 27) took 2880 ms (client timeout 1000 ms). Trace:
W20260924 19:57:40.079262 17612 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 28) took 1744 ms (client timeout 1000 ms). Trace:
W20260924 19:57:40.079627 17612 rpcz_store.cc:269] 0924 19:57:38.334864 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:38.336279 (+ 1415us) service_pool.cc:224] Handling call
0924 19:57:38.387321 (+ 51042us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:40.079202 (+1691881us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1644,"reported_tablets":24,"spinlock_wait_cycles":95819392}
I20260924 19:57:40.095003 18224 tablet_bootstrap.cc:654] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:40.061970 17613 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_14bf9267f2da4ba48354027fc92cab3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:40.104064 17613 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14bf9267f2da4ba48354027fc92cab3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:40.079514 17609 rpcz_store.cc:269] 0924 19:57:37.198347 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.297305 (+ 98958us) service_pool.cc:224] Handling call
0924 19:57:40.079045 (+2781740us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:38.164921 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:38.165431 (+ 510us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:38.165459 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:57:38.165477 (+ 18us) tablet.cc:662] Decoding operations
0924 19:57:38.166020 (+ 543us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:38.166058 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:38.166082 (+ 24us) tablet.cc:685] Acquiring locks for 4 operations
0924 19:57:38.166441 (+ 359us) tablet.cc:701] Row locks acquired
0924 19:57:38.166460 (+ 19us) write_op.cc:260] PREPARE: finished
0924 19:57:38.305058 (+138598us) write_op.cc:270] Start()
0924 19:57:38.307328 (+ 2270us) write_op.cc:276] Timestamp: P: 1790279858166501 usec, L: 0
0924 19:57:38.307352 (+ 24us) op_driver.cc:348] REPLICATION: starting
0924 19:57:38.308228 (+ 876us) log.cc:844] Serialized 1662 byte log entry
0924 19:57:38.380762 (+ 72534us) op_driver.cc:464] REPLICATION: finished
0924 19:57:38.384780 (+ 4018us) write_op.cc:301] APPLY: starting
0924 19:57:38.384845 (+ 65us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:38.385010 (+ 165us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:38.385113 (+ 103us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:38.385529 (+ 416us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:38.385695 (+ 166us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0924 19:57:38.385739 (+ 44us) write_op.cc:312] APPLY: finished
0924 19:57:38.385960 (+ 221us) log.cc:844] Serialized 41 byte log entry
0924 19:57:38.386137 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.386217 (+ 80us) write_op.cc:454] Released schema lock
0924 19:57:38.386251 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:38.386494 (+ 243us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:38.386515 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16468,"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":3834,"apply.run_cpu_time_us":1694,"apply.run_wall_time_us":1695,"num_ops":4,"prepare.queue_time_us":204213,"prepare.run_cpu_time_us":6674,"prepare.run_wall_time_us":214796,"raft.queue_time_us":14921,"raft.run_cpu_time_us":20,"raft.run_wall_time_us":14,"replication_time_us":73314,"spinlock_wait_cycles":52480,"thread_start_us":71017,"threads_started":1}]]}
I20260924 19:57:40.109764 17609 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_ee93a1803fd54a4ba6723d1e3783c004"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:40.111439 17609 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee93a1803fd54a4ba6723d1e3783c004 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:40.130946 17612 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_b278e4873a924338b9b880b50266f306" } new_table_name: "default.Table_a7d63063e7ee4493abe8c76da6d4f99d" modify_external_catalogs: true
I20260924 19:57:40.134272 17610 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet dd2f4eb8bf254c4e85b6a9072271766b
I20260924 19:57:40.138324 17610 catalog_manager.cc:6934] ReplaceTablet: tablet dd2f4eb8bf254c4e85b6a9072271766b deleted and replaced by tablet 2a1ac7e1b192465696640c02898bc39f
I20260924 19:57:40.138798 17610 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51280 (request call id 205) took 1544 ms. Trace:
I20260924 19:57:40.139207 17610 rpcz_store.cc:276] 0924 19:57:38.594571 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.868706 (+1274135us) service_pool.cc:224] Handling call
0924 19:57:40.138744 (+270038us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:40.021916 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:40.022533 (+ 617us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:40.022575 (+ 42us) write_op.cc:435] Acquired schema lock
0924 19:57:40.022595 (+ 20us) tablet.cc:662] Decoding operations
0924 19:57:40.023030 (+ 435us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:40.023077 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:40.023119 (+ 42us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:40.023471 (+ 352us) tablet.cc:701] Row locks acquired
0924 19:57:40.023496 (+ 25us) write_op.cc:260] PREPARE: finished
0924 19:57:40.023659 (+ 163us) write_op.cc:270] Start()
0924 19:57:40.023778 (+ 119us) write_op.cc:276] Timestamp: P: 1790279860023591 usec, L: 0
0924 19:57:40.023799 (+ 21us) op_driver.cc:348] REPLICATION: starting
0924 19:57:40.024536 (+ 737us) log.cc:844] Serialized 625 byte log entry
0924 19:57:40.108603 (+ 84067us) op_driver.cc:464] REPLICATION: finished
0924 19:57:40.128152 (+ 19549us) write_op.cc:301] APPLY: starting
0924 19:57:40.128220 (+ 68us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:40.128349 (+ 129us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:40.128478 (+ 129us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:40.128871 (+ 393us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:40.129117 (+ 246us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:40.129178 (+ 61us) write_op.cc:312] APPLY: finished
0924 19:57:40.129389 (+ 211us) log.cc:844] Serialized 29 byte log entry
0924 19:57:40.129566 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.129649 (+ 83us) write_op.cc:454] Released schema lock
0924 19:57:40.129689 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:40.129965 (+ 276us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.129999 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":19357,"apply.run_cpu_time_us":1789,"apply.run_wall_time_us":1788,"num_ops":2,"prepare.queue_time_us":102,"prepare.run_cpu_time_us":3054,"prepare.run_wall_time_us":4942,"raft.queue_time_us":1258,"raft.run_cpu_time_us":874,"raft.run_wall_time_us":875,"replication_time_us":84679}]]}
I20260924 19:57:40.148522 17796 tablet_service.cc:1558] Processing DeleteTablet for tablet dd2f4eb8bf254c4e85b6a9072271766b with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1ac7e1b192465696640c02898bc39f) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:40.169493 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 221539us
I20260924 19:57:40.176685 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e7c5b28fea4dbda802702c8b85ec54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:40.190606 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet dd2f4eb8bf254c4e85b6a9072271766b with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1ac7e1b192465696640c02898bc39f) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:40.203439 18777 rpcz_store.cc:276] 0924 19:57:38.125593 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:38.125707 (+ 114us) service_pool.cc:224] Handling call
0924 19:57:39.406679 (+1280972us) ts_tablet_manager.cc:1151] Deleting tablet 2cf62a73c49245198d65011875148dc3
0924 19:57:39.406724 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:39.726137 (+319413us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:39.964460 (+238323us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:40.044775 (+ 80315us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1280427}
I20260924 19:57:40.050937 18873 raft_consensus.cc:493] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:40.056429 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e6d926ff7c24901a15b0c7b2239fbad" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:40.207510 18873 raft_consensus.cc:515] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:40.208905 17753 leader_election.cc:343] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:40.142838 18082 tablet_service.cc:1558] Processing DeleteTablet for tablet dd2f4eb8bf254c4e85b6a9072271766b with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1ac7e1b192465696640c02898bc39f) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:40.151361 18115 raft_consensus.cc:3060] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:40.216655 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6bd442f16492495f84bb8363b9bade97" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:40.217375 17815 raft_consensus.cc:2468] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:40.218888 18051 leader_election.cc:304] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:40.219739 18746 raft_consensus.cc:2804] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:40.220099 18746 raft_consensus.cc:493] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:40.220733 18746 raft_consensus.cc:3060] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:40.226282 18780 consensus_queue.cc:1048] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:40.230419 18223 tablet_bootstrap.cc:492] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:40.229518 18873 leader_election.cc:290] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:40.267211 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 167) took 2140 ms. Trace:
I20260924 19:57:40.267880 18777 rpcz_store.cc:276] 0924 19:57:38.126639 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:38.126738 (+ 99us) service_pool.cc:224] Handling call
0924 19:57:40.207023 (+2080285us) ts_tablet_manager.cc:1151] Deleting tablet 57008266b80345e0a3f8e60bd7e2d62b
0924 19:57:40.207149 (+ 126us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:40.267085 (+ 59936us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2079639}
I20260924 19:57:40.273518 18777 tablet_replica.cc:333] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:40.279979 18777 raft_consensus.cc:2243] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:40.287323 18777 raft_consensus.cc:2272] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:40.286722 18223 tablet_bootstrap.cc:654] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:40.296545 18777 ts_tablet_manager.cc:1916] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:40.316388 18231 ts_tablet_manager.cc:1929] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:40.316946 18231 log.cc:1199] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/41bdb846ca19425b882434df0167479e
I20260924 19:57:40.318142 18231 ts_tablet_manager.cc:1950] T 41bdb846ca19425b882434df0167479e P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:40.320052 18910 tablet_replica.cc:333] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
W20260924 19:57:40.323098 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 354298us
I20260924 19:57:40.373275 18224 tablet_bootstrap.cc:492] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:40.374014 18224 ts_tablet_manager.cc:1403] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.304s user 0.054s sys 0.053s
I20260924 19:57:40.376608 18224 raft_consensus.cc:359] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.242846 18746 raft_consensus.cc:515] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.378175 18224 raft_consensus.cc:385] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:40.379406 18746 leader_election.cc:290] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:40.387252 18224 raft_consensus.cc:740] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:40.388108 18224 consensus_queue.cc:260] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.396965 18224 ts_tablet_manager.cc:1434] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
W20260924 19:57:40.142357 17599 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6bd442f16492495f84bb8363b9bade97 consensus state has no leader
I20260924 19:57:40.399784 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 2cf62a73c49245198d65011875148dc3 (table default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265]) successfully deleted
W20260924 19:57:40.400465 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 57008266b80345e0a3f8e60bd7e2d62b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57008266b80345e0a3f8e60bd7e2d62b already in progress: creating tablet
I20260924 19:57:40.419988 18780 raft_consensus.cc:493] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:40.420475 18780 raft_consensus.cc:515] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.422573 17613 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 208) took 1626 ms. Trace:
I20260924 19:57:40.422683 18780 leader_election.cc:290] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:40.422894 17613 rpcz_store.cc:276] 0924 19:57:38.796209 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.061619 (+1265410us) service_pool.cc:224] Handling call
0924 19:57:40.133739 (+ 72120us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:40.135893 (+ 2154us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:40.421763 (+285870us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:40.422240 (+ 477us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:40.422511 (+ 271us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:40.154221 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:40.154707 (+ 486us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:40.154737 (+ 30us) write_op.cc:435] Acquired schema lock
0924 19:57:40.154754 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:40.155085 (+ 331us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:40.155131 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:40.155173 (+ 42us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:40.155404 (+ 231us) tablet.cc:701] Row locks acquired
0924 19:57:40.155417 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:57:40.155546 (+ 129us) write_op.cc:270] Start()
0924 19:57:40.155639 (+ 93us) write_op.cc:276] Timestamp: P: 1790279860155474 usec, L: 0
0924 19:57:40.155658 (+ 19us) op_driver.cc:348] REPLICATION: starting
0924 19:57:40.156226 (+ 568us) log.cc:844] Serialized 433 byte log entry
0924 19:57:40.407443 (+251217us) op_driver.cc:464] REPLICATION: finished
0924 19:57:40.419873 (+ 12430us) write_op.cc:301] APPLY: starting
0924 19:57:40.419948 (+ 75us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:40.420066 (+ 118us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:40.420162 (+ 96us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:40.420779 (+ 617us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:40.420935 (+ 156us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:40.420979 (+ 44us) write_op.cc:312] APPLY: finished
0924 19:57:40.421284 (+ 305us) log.cc:844] Serialized 29 byte log entry
0924 19:57:40.421457 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.421526 (+ 69us) write_op.cc:454] Released schema lock
0924 19:57:40.421561 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":12238,"num_ops":2,"prepare.queue_time_us":17118,"prepare.run_cpu_time_us":2382,"prepare.run_wall_time_us":3235,"raft.queue_time_us":957,"raft.run_cpu_time_us":638,"raft.run_wall_time_us":638,"replication_time_us":251648}]]}
I20260924 19:57:40.428687 17613 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_2b157ac766d244e081168b817296bd4b" } modify_external_catalogs: true
I20260924 19:57:40.429337 17613 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_2b157ac766d244e081168b817296bd4b" } modify_external_catalogs: true
I20260924 19:57:40.446240 17610 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_fca275d61c914e72b792ca0bb116d0bf" } modify_external_catalogs: true
I20260924 19:57:40.447703 18224 tablet_bootstrap.cc:492] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:40.457244 18224 tablet_bootstrap.cc:654] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:40.479985 18220 tablet_bootstrap.cc:492] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:40.480480 18220 ts_tablet_manager.cc:1403] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.747s user 0.016s sys 0.002s
I20260924 19:57:40.482295 17610 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_fca275d61c914e72b792ca0bb116d0bf" } modify_external_catalogs: true
I20260924 19:57:40.484613 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 158) took 2965 ms. Trace:
I20260924 19:57:40.397955 18959 consensus_queue.cc:1048] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:40.431491 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df3dcf4ddcf4efa9aa047da56dfb7a7" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:40.436321 18777 ts_tablet_manager.cc:1929] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:40.539546 18777 log.cc:1199] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/c884f12860164f12bbcd0d2f87067013
I20260924 19:57:40.540325 18777 ts_tablet_manager.cc:1950] T c884f12860164f12bbcd0d2f87067013 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
W20260924 19:57:40.480988 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 139419us
I20260924 19:57:40.486128 18050 leader_election.cc:304] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:40.542385 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 56337.8us
I20260924 19:57:40.543285 18959 raft_consensus.cc:2749] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:40.543618 18050 leader_election.cc:304] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:40.544613 18959 raft_consensus.cc:2749] T c0feecd2fdb4409f86f6a5fbfbab3537 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:40.550105 18231 rpcz_store.cc:276] 0924 19:57:37.518912 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:37.519994 (+ 1082us) service_pool.cc:224] Handling call
0924 19:57:39.540086 (+2020092us) ts_tablet_manager.cc:1151] Deleting tablet 41bdb846ca19425b882434df0167479e
0924 19:57:39.540130 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:39.851580 (+311450us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:40.316228 (+464648us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:40.484547 (+168319us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2019570}
I20260924 19:57:40.551386 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 41bdb846ca19425b882434df0167479e (table default.Table_0bb01b569e3c41a4a53d6eb7817aaade [id=ad9f72129d8343e0932f8035a6554aab]) successfully deleted
I20260924 19:57:40.520174 18224 tablet_bootstrap.cc:492] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:40.544483 18970 raft_consensus.cc:493] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:40.552491 18224 ts_tablet_manager.cc:1403] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.105s user 0.006s sys 0.006s
I20260924 19:57:40.552755 18970 raft_consensus.cc:515] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:40.546154 17753 leader_election.cc:343] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:40.554327 17606 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 209) took 1678 ms. Trace:
I20260924 19:57:40.554997 18970 leader_election.cc:290] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:40.555298 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6bd442f16492495f84bb8363b9bade97" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:40.555861 17815 raft_consensus.cc:3060] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:40.554666 17606 rpcz_store.cc:276] 0924 19:57:38.875856 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.064032 (+1188176us) service_pool.cc:224] Handling call
0924 19:57:40.133577 (+ 69545us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:40.152996 (+ 19419us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:40.553694 (+400698us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:40.553995 (+ 301us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:40.554269 (+ 274us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:40.157575 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:40.158052 (+ 477us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:40.158079 (+ 27us) write_op.cc:435] Acquired schema lock
0924 19:57:40.158095 (+ 16us) tablet.cc:662] Decoding operations
0924 19:57:40.158691 (+ 596us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:40.158738 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:40.158767 (+ 29us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:40.159035 (+ 268us) tablet.cc:701] Row locks acquired
0924 19:57:40.159049 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:40.159206 (+ 157us) write_op.cc:270] Start()
0924 19:57:40.159302 (+ 96us) write_op.cc:276] Timestamp: P: 1790279860159107 usec, L: 0
0924 19:57:40.159318 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:40.160004 (+ 686us) log.cc:844] Serialized 433 byte log entry
0924 19:57:40.550515 (+390511us) op_driver.cc:464] REPLICATION: finished
0924 19:57:40.552078 (+ 1563us) write_op.cc:301] APPLY: starting
0924 19:57:40.552134 (+ 56us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:40.552240 (+ 106us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:40.552337 (+ 97us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:40.552810 (+ 473us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:40.552956 (+ 146us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:40.553012 (+ 56us) write_op.cc:312] APPLY: finished
0924 19:57:40.553221 (+ 209us) log.cc:844] Serialized 29 byte log entry
0924 19:57:40.553393 (+ 172us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.553461 (+ 68us) write_op.cc:454] Released schema lock
0924 19:57:40.553498 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:40.553730 (+ 232us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.553751 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1396,"apply.run_cpu_time_us":1588,"apply.run_wall_time_us":1627,"num_ops":2,"prepare.queue_time_us":3235,"prepare.run_cpu_time_us":2540,"prepare.run_wall_time_us":2915,"raft.queue_time_us":85,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":391093}]]}
I20260924 19:57:40.568699 17815 raft_consensus.cc:2468] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:40.574438 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:40.576112 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92c0daa7d67e4aa390e334c669adda76" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:40.582672 17753 leader_election.cc:343] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:40.588097 18231 tablet_replica.cc:333] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:40.591580 18220 raft_consensus.cc:359] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.592391 18220 raft_consensus.cc:385] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:40.592680 18220 raft_consensus.cc:740] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:40.593348 18220 consensus_queue.cc:260] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.597590 18231 raft_consensus.cc:2243] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:40.598129 18231 raft_consensus.cc:2272] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:40.603885 18231 ts_tablet_manager.cc:1916] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:40.576931 17608 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 206) took 1982 ms. Trace:
I20260924 19:57:40.578800 17609 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 210) took 1535 ms. Trace:
I20260924 19:57:40.615360 17609 rpcz_store.cc:276] 0924 19:57:39.043614 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.109396 (+1065782us) service_pool.cc:224] Handling call
0924 19:57:40.140744 (+ 31348us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:40.142131 (+ 1387us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:40.578226 (+436095us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:40.578507 (+ 281us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:40.578747 (+ 240us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:40.234524 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:40.235356 (+ 832us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:40.235397 (+ 41us) write_op.cc:435] Acquired schema lock
0924 19:57:40.235424 (+ 27us) tablet.cc:662] Decoding operations
0924 19:57:40.235874 (+ 450us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:40.235933 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:40.235982 (+ 49us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:40.236301 (+ 319us) tablet.cc:701] Row locks acquired
0924 19:57:40.236319 (+ 18us) write_op.cc:260] PREPARE: finished
0924 19:57:40.236483 (+ 164us) write_op.cc:270] Start()
0924 19:57:40.236602 (+ 119us) write_op.cc:276] Timestamp: P: 1790279860236393 usec, L: 0
0924 19:57:40.236626 (+ 24us) op_driver.cc:348] REPLICATION: starting
0924 19:57:40.237339 (+ 713us) log.cc:844] Serialized 433 byte log entry
0924 19:57:40.569819 (+332480us) op_driver.cc:464] REPLICATION: finished
0924 19:57:40.576609 (+ 6790us) write_op.cc:301] APPLY: starting
0924 19:57:40.576656 (+ 47us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:40.576754 (+ 98us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:40.576847 (+ 93us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:40.577220 (+ 373us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:40.577345 (+ 125us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:40.577382 (+ 37us) write_op.cc:312] APPLY: finished
0924 19:57:40.577803 (+ 421us) log.cc:844] Serialized 29 byte log entry
0924 19:57:40.577960 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.578019 (+ 59us) write_op.cc:454] Released schema lock
0924 19:57:40.578049 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6637,"apply.run_cpu_time_us":1371,"apply.run_wall_time_us":1627,"mutex_wait_us":27020,"num_ops":2,"prepare.queue_time_us":91234,"prepare.run_cpu_time_us":3078,"prepare.run_wall_time_us":39845,"raft.queue_time_us":256,"raft.run_cpu_time_us":16,"raft.run_wall_time_us":16,"replication_time_us":333111,"thread_start_us":27302,"threads_started":1}]]}
I20260924 19:57:40.618271 18910 raft_consensus.cc:2243] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:40.612457 17608 rpcz_store.cc:276] 0924 19:57:38.594834 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.045788 (+1450954us) service_pool.cc:224] Handling call
0924 19:57:40.137070 (+ 91282us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:40.139672 (+ 2602us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:40.576294 (+436622us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:40.576571 (+ 277us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:40.576873 (+ 302us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:40.179335 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:40.186107 (+ 6772us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:40.186386 (+ 279us) write_op.cc:435] Acquired schema lock
0924 19:57:40.186509 (+ 123us) tablet.cc:662] Decoding operations
0924 19:57:40.192262 (+ 5753us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:40.192685 (+ 423us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:40.192994 (+ 309us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:40.196088 (+ 3094us) tablet.cc:701] Row locks acquired
0924 19:57:40.196116 (+ 28us) write_op.cc:260] PREPARE: finished
0924 19:57:40.199917 (+ 3801us) write_op.cc:270] Start()
0924 19:57:40.200925 (+ 1008us) write_op.cc:276] Timestamp: P: 1790279860198470 usec, L: 0
0924 19:57:40.201143 (+ 218us) op_driver.cc:348] REPLICATION: starting
0924 19:57:40.233731 (+ 32588us) log.cc:844] Serialized 433 byte log entry
0924 19:57:40.569540 (+335809us) op_driver.cc:464] REPLICATION: finished
0924 19:57:40.569854 (+ 314us) write_op.cc:301] APPLY: starting
0924 19:57:40.569921 (+ 67us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:40.570021 (+ 100us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:40.570116 (+ 95us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:40.575431 (+ 5315us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:40.575577 (+ 146us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:40.575630 (+ 53us) write_op.cc:312] APPLY: finished
0924 19:57:40.575811 (+ 181us) log.cc:844] Serialized 29 byte log entry
0924 19:57:40.575976 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.576040 (+ 64us) write_op.cc:454] Released schema lock
0924 19:57:40.576072 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:40.576329 (+ 257us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.576355 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":126,"apply.run_cpu_time_us":4632,"apply.run_wall_time_us":6453,"mutex_wait_us":28740,"num_ops":2,"prepare.queue_time_us":37198,"prepare.run_cpu_time_us":4508,"prepare.run_wall_time_us":55913,"raft.queue_time_us":122,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":16,"replication_time_us":368285}]]}
I20260924 19:57:40.627339 17608 master_service.cc:960] ReplaceTablet: received request to replace tablet 83ee43b6dc064dc7b320691d52e57ea7 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:40.634339 18777 tablet_replica.cc:333] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:40.599274 17606 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_6c2fccb54ef24d088cd2ec174a8e20e9" } new_table_name: "default.Table_454683a3d79d4988b41292f7776df549" modify_external_catalogs: true
I20260924 19:57:40.637185 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_b278e4873a924338b9b880b50266f306 [id=be19c62729a34ae1a15966662303a869]). Expected version 1 got 0
I20260924 19:57:40.637414 18777 raft_consensus.cc:2243] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:40.638031 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:40.638621 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:40.639511 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_7b49c63ad79c4132afbbbb9e1f2d4ca0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 3 got 2
I20260924 19:57:40.640307 17614 catalog_manager.cc:5838] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P c4f6e3a7b06148918d230c77ab0c8202 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
I20260924 19:57:40.515751 18223 tablet_bootstrap.cc:492] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:40.642159 18223 ts_tablet_manager.cc:1403] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.602s user 0.016s sys 0.015s
I20260924 19:57:40.594686 18224 raft_consensus.cc:359] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.655432 18224 raft_consensus.cc:385] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:40.655827 18224 raft_consensus.cc:740] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:40.656900 18224 consensus_queue.cc:260] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.659077 18224 ts_tablet_manager.cc:1434] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.106s user 0.006s sys 0.000s
I20260924 19:57:40.664937 18909 raft_consensus.cc:493] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:40.665462 18909 raft_consensus.cc:515] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.667704 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e6d926ff7c24901a15b0c7b2239fbad" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:40.687515 18777 raft_consensus.cc:2272] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:40.688265 18224 tablet_bootstrap.cc:492] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:40.691795 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]). Expected version 2 got 1
I20260924 19:57:40.697997 18777 ts_tablet_manager.cc:1916] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:40.702448 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
W20260924 19:57:40.709956 18053 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.17.2.190:46309, user_credentials={real_user=slave}} blocked reactor thread for 51040.8us
I20260924 19:57:40.711067 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet c884f12860164f12bbcd0d2f87067013 (table default.Table_5b3a0213ccbb4622a13a68e24d12af4e [id=54f05b78c3324978b2635be4bd969f66]) successfully deleted
I20260924 19:57:40.715797 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92c0daa7d67e4aa390e334c669adda76" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:40.715929 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df3dcf4ddcf4efa9aa047da56dfb7a7" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:40.720497 18963 consensus_queue.cc:1048] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:40.720744 18051 leader_election.cc:304] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
W20260924 19:57:40.692065 17753 consensus_peers.cc:597] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c884f12860164f12bbcd0d2f87067013. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:40.722289 18963 consensus_queue.cc:1048] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:40.723315 18746 raft_consensus.cc:2804] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
W20260924 19:57:40.723353 17753 leader_election.cc:343] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:40.723762 18746 raft_consensus.cc:697] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:40.724565 18746 consensus_queue.cc:237] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.725217 18910 raft_consensus.cc:2272] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:40.726017 18909 leader_election.cc:290] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:40.730621 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6bd442f16492495f84bb8363b9bade97" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:40.732044 17969 raft_consensus.cc:3060] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:40.737092 18910 ts_tablet_manager.cc:1916] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:40.741655 17602 consensus_peers.cc:597] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e -> Peer 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309): Couldn't send request to peer 3e059bbe4dac4178bccb3be48cb1118d. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.190:46309: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260924 19:57:40.742697 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6bd442f16492495f84bb8363b9bade97" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:40.752599 18220 ts_tablet_manager.cc:1434] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.272s user 0.007s sys 0.001s
I20260924 19:57:40.700891 18223 raft_consensus.cc:359] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.753366 18223 raft_consensus.cc:385] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:40.753785 18223 raft_consensus.cc:740] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
W20260924 19:57:40.755034 17606 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 78801b8d0b664217a74b20160ba2d8ac consensus state has no leader
I20260924 19:57:40.736985 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbb82914df65416aaf47db8e507afb40" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:40.755976 17816 raft_consensus.cc:2393] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:40.755738 17753 leader_election.cc:304] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:40.760406 18912 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s user 0.006s sys 0.022s
I20260924 19:57:40.771297 18966 raft_consensus.cc:2749] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:40.772660 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 49326.3us
W20260924 19:57:40.783234 17753 leader_election.cc:343] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:40.754621 18223 consensus_queue.cc:260] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:40.787464 18220 tablet_bootstrap.cc:492] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:40.796772 18223 ts_tablet_manager.cc:1434] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.154s user 0.006s sys 0.000s
I20260924 19:57:40.806742 17753 leader_election.cc:304] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:40.816367 17972 raft_consensus.cc:3060] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:40.798532 18050 leader_election.cc:343] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:40.833833 18050 leader_election.cc:343] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:40.706627 17614 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:40.800618 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dbb82914df65416aaf47db8e507afb40" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:40.836443 18746 consensus_queue.cc:1048] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:40.823545 18115 raft_consensus.cc:3060] T fd6e656a653a43d1843a91fee813237d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:40.856559 18050 consensus_peers.cc:597] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:40.864091 18223 tablet_bootstrap.cc:492] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:40.867713 18780 raft_consensus.cc:2749] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:40.845491 18224 tablet_bootstrap.cc:700] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202: Time spent opening tablet: real 0.141s user 0.001s sys 0.001s
I20260924 19:57:40.869230 18224 tablet_bootstrap.cc:654] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:40.870833 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 87653.3us
W20260924 19:57:40.872321 17753 leader_election.cc:343] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:40.872687 17753 leader_election.cc:304] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:40.873377 18780 raft_consensus.cc:2749] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:40.873965 18780 consensus_queue.cc:1048] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:40.873150 18992 log.cc:927] Time spent T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260924 19:57:40.902032 18223 tablet_bootstrap.cc:654] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:40.930235 17607 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]). Expected version 2 got 1
I20260924 19:57:40.930848 17608 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 83ee43b6dc064dc7b320691d52e57ea7
I20260924 19:57:40.933341 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet 83ee43b6dc064dc7b320691d52e57ea7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c1afbfc7de7480ab4be40680a3e3ada) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:40.934453 17608 catalog_manager.cc:6934] ReplaceTablet: tablet 83ee43b6dc064dc7b320691d52e57ea7 deleted and replaced by tablet 7c1afbfc7de7480ab4be40680a3e3ada
I20260924 19:57:40.943928 18082 tablet_service.cc:1558] Processing DeleteTablet for tablet 83ee43b6dc064dc7b320691d52e57ea7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c1afbfc7de7480ab4be40680a3e3ada) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:40.945358 17949 tablet_service.cc:1558] Processing DeleteTablet for tablet 83ee43b6dc064dc7b320691d52e57ea7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c1afbfc7de7480ab4be40680a3e3ada) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:40.953514 18746 consensus_queue.cc:1048] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:40.960469 18993 tablet.cc:1751] T 954178b8cc1e434085337f66b0e85672 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260924 19:57:40.964711 18999 tablet.cc:1751] T 954178b8cc1e434085337f66b0e85672 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260924 19:57:40.966305 17607 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_7b49c63ad79c4132afbbbb9e1f2d4ca0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 3 got 2
I20260924 19:57:40.968457 17607 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:40.970240 17607 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:40.970391 18777 ts_tablet_manager.cc:1929] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:40.970769 18777 log.cc:1199] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/8a524bad1c4040e58ad84fcf5a4fc47d
W20260924 19:57:40.966748 17905 consensus_peers.cc:597] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260924 19:57:40.973683 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 96700.4us
I20260924 19:57:40.979668 17945 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51332 (request call id 171) took 1036 ms. Trace:
W20260924 19:57:40.979779 17906 consensus_peers.cc:597] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a765abd702644f079bddffb3fc4238ef. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:40.980158 17945 rpcz_store.cc:276] 0924 19:57:39.943638 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.984811 (+ 41173us) service_pool.cc:224] Handling call
0924 19:57:40.012031 (+ 27220us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:40.012220 (+ 189us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:40.627475 (+615255us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:40.979213 (+351738us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:40.979613 (+ 400us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:40.985368 18777 ts_tablet_manager.cc:1950] T 8a524bad1c4040e58ad84fcf5a4fc47d P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:40.986536 17905 leader_election.cc:304] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:40.993382 18220 tablet_bootstrap.cc:654] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:40.995820 18231 ts_tablet_manager.cc:1929] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:40.996214 18231 log.cc:1199] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/2cf62a73c49245198d65011875148dc3
I20260924 19:57:40.996835 18231 ts_tablet_manager.cc:1950] T 2cf62a73c49245198d65011875148dc3 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:40.997534 18909 raft_consensus.cc:2749] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:40.998709 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 166) took 2888 ms. Trace:
I20260924 19:57:41.001997 19000 tablet.cc:1751] T 954178b8cc1e434085337f66b0e85672 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260924 19:57:41.004755 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 174) took 1140 ms. Trace:
I20260924 19:57:41.005105 18777 rpcz_store.cc:276] 0924 19:57:39.864388 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.867251 (+ 2863us) service_pool.cc:224] Handling call
0924 19:57:40.541440 (+674189us) ts_tablet_manager.cc:1151] Deleting tablet 8a524bad1c4040e58ad84fcf5a4fc47d
0924 19:57:40.541476 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:40.634256 (+ 92780us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b5c00032c0c. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb843b60151 00007fb843d06229 00007fb844852914 00007fb844883960 00007fb844869e6d 00007fb844869e19 00007fb844869da9 00007fb844869d71 00007fb844868f9d 00000000005bd4e4 00000000005bd319
0924 19:57:40.879946 (+245690us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:40.970267 (+ 90321us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.004681 (+ 34414us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49525,"spinlock_wait_cycles":11328896,"tablet-delete.queue_time_us":673669}
I20260924 19:57:41.013249 18998 tablet.cc:1751] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:41.017067 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 175) took 1091 ms. Trace:
I20260924 19:57:41.017428 18777 rpcz_store.cc:276] 0924 19:57:39.925285 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.925392 (+ 107us) service_pool.cc:224] Handling call
0924 19:57:41.016801 (+1091409us) ts_tablet_manager.cc:1151] Deleting tablet 24f84032bca24a7ca9da65f0bbceb0d5
0924 19:57:41.016842 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.017008 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090862}
I20260924 19:57:41.022271 18223 tablet_bootstrap.cc:492] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:41.022696 18223 ts_tablet_manager.cc:1403] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.159s user 0.006s sys 0.010s
I20260924 19:57:41.025171 18223 raft_consensus.cc:359] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.025790 18223 raft_consensus.cc:385] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:41.026038 18223 raft_consensus.cc:740] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:41.032791 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 176) took 1105 ms. Trace:
I20260924 19:57:41.033175 18777 rpcz_store.cc:276] 0924 19:57:39.926953 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.927043 (+ 90us) service_pool.cc:224] Handling call
0924 19:57:41.031379 (+1104336us) ts_tablet_manager.cc:1151] Deleting tablet 349c6e9413bb41268d0c41d549203e2a
0924 19:57:41.031421 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.031570 (+ 149us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1103819}
I20260924 19:57:41.034739 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 177) took 1105 ms. Trace:
I20260924 19:57:41.035060 18777 rpcz_store.cc:276] 0924 19:57:39.929007 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.929108 (+ 101us) service_pool.cc:224] Handling call
0924 19:57:41.034480 (+1105372us) ts_tablet_manager.cc:1151] Deleting tablet 151247d6b5254cd884c58db9311ac04a
0924 19:57:41.034527 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.034685 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1104919}
I20260924 19:57:41.036958 18777 tablet_replica.cc:333] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:41.037743 18777 raft_consensus.cc:2243] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:41.038599 18777 raft_consensus.cc:2272] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:41.041501 18777 ts_tablet_manager.cc:1916] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:41.045039 18951 consensus_queue.cc:1048] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:41.050079 18999 tablet.cc:1751] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:41.026624 18223 consensus_queue.cc:260] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.076272 18223 ts_tablet_manager.cc:1434] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.053s user 0.007s sys 0.000s
I20260924 19:57:41.080201 18996 tablet.cc:1751] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:41.088996 18223 tablet_bootstrap.cc:492] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:41.091348 18224 tablet_bootstrap.cc:492] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:41.091223 18778 consensus_queue.cc:1048] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:41.097409 18224 ts_tablet_manager.cc:1403] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.437s user 0.009s sys 0.009s
I20260924 19:57:41.071847 17967 raft_consensus.cc:3060] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:41.101918 18910 ts_tablet_manager.cc:1929] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:41.102490 18910 log.cc:1199] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/41bdb846ca19425b882434df0167479e
I20260924 19:57:41.104243 18910 ts_tablet_manager.cc:1950] T 41bdb846ca19425b882434df0167479e P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:41.109191 18223 tablet_bootstrap.cc:654] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:41.111517 18224 raft_consensus.cc:359] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.116593 18224 raft_consensus.cc:385] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:41.118770 18224 raft_consensus.cc:740] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:41.119912 18224 consensus_queue.cc:260] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.128196 18224 ts_tablet_manager.cc:1434] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.030s user 0.001s sys 0.004s
I20260924 19:57:41.115317 18993 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51332 (request call id 172) took 1162 ms. Trace:
I20260924 19:57:41.132846 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 178) took 1186 ms. Trace:
I20260924 19:57:41.138295 18777 ts_tablet_manager.cc:1929] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:41.133334 18910 rpcz_store.cc:276] 0924 19:57:39.946140 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.979079 (+ 32939us) service_pool.cc:224] Handling call
0924 19:57:40.251537 (+272458us) ts_tablet_manager.cc:1151] Deleting tablet 41bdb846ca19425b882434df0167479e
0924 19:57:40.254156 (+ 2619us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.003944 (+749788us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.101765 (+ 97821us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.132756 (+ 30991us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":387310,"tablet-delete.queue_time_us":190760,"thread_start_us":30375,"threads_started":1}
I20260924 19:57:41.132107 18993 rpcz_store.cc:276] 0924 19:57:39.944056 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.988588 (+ 44532us) service_pool.cc:224] Handling call
0924 19:57:41.105985 (+1117397us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:40.011255 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0924 19:57:40.011750 (+ 495us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0924 19:57:40.011819 (+ 69us) alter_schema_op.cc:66] Acquired schema lock
0924 19:57:40.011916 (+ 97us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0924 19:57:40.012064 (+ 148us) alter_schema_op.cc:128] START. Timestamp: 1790279860011921
0924 19:57:40.012104 (+ 40us) op_driver.cc:348] REPLICATION: starting
0924 19:57:40.012954 (+ 850us) log.cc:844] Serialized 173 byte log entry
0924 19:57:40.822769 (+809815us) op_driver.cc:464] REPLICATION: finished
0924 19:57:40.960257 (+137488us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0924 19:57:40.995283 (+ 35026us) log.cc:844] Serialized 5 byte log entry
0924 19:57:41.104992 (+109709us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.105721 (+ 729us) log.cc:844] Serialized 14 byte log entry
0924 19:57:41.105822 (+ 101us) alter_schema_op.cc:73] Released schema lock
0924 19:57:41.105851 (+ 29us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":137265,"prepare.queue_time_us":20556,"prepare.run_cpu_time_us":28195,"prepare.run_wall_time_us":752329,"raft.queue_time_us":363,"raft.run_cpu_time_us":814,"raft.run_wall_time_us":813,"replication_time_us":810525,"thread_start_us":1046352,"threads_started":3,"wal-append.queue_time_us":462786}]]}
I20260924 19:57:41.150025 18910 tablet_replica.cc:333] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:41.151685 18996 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51332 (request call id 169) took 1208 ms. Trace:
I20260924 19:57:41.158452 18220 tablet_bootstrap.cc:492] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:41.158877 18220 ts_tablet_manager.cc:1403] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.405s user 0.009s sys 0.006s
I20260924 19:57:41.161430 18224 tablet_bootstrap.cc:492] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:41.006685 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]) successfully deleted
W20260924 19:57:41.172457 17599 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 165785us
I20260924 19:57:41.173121 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 41bdb846ca19425b882434df0167479e (table default.Table_0bb01b569e3c41a4a53d6eb7817aaade [id=ad9f72129d8343e0932f8035a6554aab]) successfully deleted
W20260924 19:57:41.173990 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 24f84032bca24a7ca9da65f0bbceb0d5 because the tablet was not found. No further retry: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5
W20260924 19:57:41.174623 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 349c6e9413bb41268d0c41d549203e2a because the tablet was not found. No further retry: Not found: Tablet not found: 349c6e9413bb41268d0c41d549203e2a
W20260924 19:57:41.175309 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 151247d6b5254cd884c58db9311ac04a because the tablet was not found. No further retry: Not found: Tablet not found: 151247d6b5254cd884c58db9311ac04a
I20260924 19:57:41.177074 17607 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
I20260924 19:57:41.178658 17607 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:41.181187 19015 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.190:46309 every 8 connection(s)
I20260924 19:57:41.184935 18224 tablet_bootstrap.cc:654] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:41.191931 18777 log.cc:1199] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/d970cc92a1134a98a613d198a319cb01
I20260924 19:57:41.177085 18223 tablet_bootstrap.cc:492] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:41.192710 18223 ts_tablet_manager.cc:1403] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.104s user 0.011s sys 0.000s
I20260924 19:57:41.195530 18223 raft_consensus.cc:359] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.196337 18223 raft_consensus.cc:385] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:41.196640 18223 raft_consensus.cc:740] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:41.197489 18223 consensus_queue.cc:260] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.199678 18223 ts_tablet_manager.cc:1434] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260924 19:57:41.170050 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 113333us
I20260924 19:57:41.208515 18777 ts_tablet_manager.cc:1950] T d970cc92a1134a98a613d198a319cb01 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:41.209584 18320 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.190:46309
I20260924 19:57:41.210934 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 179) took 1275 ms. Trace:
I20260924 19:57:41.211439 18777 rpcz_store.cc:276] 0924 19:57:39.935931 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.936023 (+ 92us) service_pool.cc:224] Handling call
0924 19:57:41.036788 (+1100765us) ts_tablet_manager.cc:1151] Deleting tablet d970cc92a1134a98a613d198a319cb01
0924 19:57:41.036842 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.106061 (+ 69219us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.138162 (+ 32101us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.210870 (+ 72708us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1100211}
I20260924 19:57:41.215205 18220 raft_consensus.cc:359] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.215988 18220 raft_consensus.cc:385] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:41.215847 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet d970cc92a1134a98a613d198a319cb01 (table default.Table_a50429edeaf645668bc84244412bb359 [id=1853f611de514a55afd7a68d742503b8]) successfully deleted
I20260924 19:57:41.216261 18220 raft_consensus.cc:740] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
W20260924 19:57:41.216583 19013 log.cc:927] Time spent T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260924 19:57:41.217032 18220 consensus_queue.cc:260] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.226255 18932 consensus_queue.cc:1048] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:41.228535 18223 tablet_bootstrap.cc:492] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:41.150960 18910 raft_consensus.cc:2243] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:41.234323 18223 tablet_bootstrap.cc:654] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:41.152050 18996 rpcz_store.cc:276] 0924 19:57:39.942856 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.991673 (+ 48817us) service_pool.cc:224] Handling call
0924 19:57:41.151628 (+1159955us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:40.254740 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0924 19:57:40.331359 (+ 76619us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0924 19:57:40.331404 (+ 45us) alter_schema_op.cc:66] Acquired schema lock
0924 19:57:40.331506 (+ 102us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0924 19:57:40.331662 (+ 156us) alter_schema_op.cc:128] START. Timestamp: 1790279860331496
0924 19:57:40.331684 (+ 22us) op_driver.cc:348] REPLICATION: starting
0924 19:57:40.332360 (+ 676us) log.cc:844] Serialized 173 byte log entry
0924 19:57:40.915592 (+583232us) op_driver.cc:464] REPLICATION: finished
0924 19:57:41.079672 (+164080us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0924 19:57:41.081492 (+ 1820us) log.cc:844] Serialized 5 byte log entry
0924 19:57:41.146518 (+ 65026us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.147272 (+ 754us) log.cc:844] Serialized 14 byte log entry
0924 19:57:41.151335 (+ 4063us) alter_schema_op.cc:73] Released schema lock
0924 19:57:41.151379 (+ 44us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":163895,"mutex_wait_us":53230,"prepare.queue_time_us":253911,"prepare.run_cpu_time_us":9067,"prepare.run_wall_time_us":474376,"raft.queue_time_us":73629,"raft.run_cpu_time_us":1713,"raft.run_wall_time_us":12787,"replication_time_us":583782,"thread_start_us":449789,"threads_started":3,"wal-append.queue_time_us":444098}]]}
I20260924 19:57:41.241349 18970 raft_consensus.cc:493] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:41.241896 18970 raft_consensus.cc:515] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.247475 18970 leader_election.cc:290] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:41.249109 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274f3decdc7e42c0917bf28151c4da71" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:41.249713 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274f3decdc7e42c0917bf28151c4da71" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:41.250633 17753 leader_election.cc:343] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:41.251780 17753 leader_election.cc:343] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:41.252269 17753 leader_election.cc:304] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:41.253005 18970 raft_consensus.cc:2749] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:41.240139 18220 ts_tablet_manager.cc:1434] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.081s user 0.004s sys 0.003s
I20260924 19:57:41.242774 18910 raft_consensus.cc:2272] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:40.999006 18231 rpcz_store.cc:276] 0924 19:57:38.110709 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:38.113802 (+ 3093us) service_pool.cc:224] Handling call
0924 19:57:40.587942 (+2474140us) ts_tablet_manager.cc:1151] Deleting tablet 2cf62a73c49245198d65011875148dc3
0924 19:57:40.587989 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:40.709427 (+121438us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:40.995681 (+286254us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:40.998657 (+ 2976us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8647,"tablet-delete.queue_time_us":2473601}
I20260924 19:57:41.257797 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 180) took 1321 ms. Trace:
I20260924 19:57:41.258008 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 2cf62a73c49245198d65011875148dc3 (table default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265]) successfully deleted
I20260924 19:57:41.258282 18777 rpcz_store.cc:276] 0924 19:57:39.936674 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.936764 (+ 90us) service_pool.cc:224] Handling call
0924 19:57:41.257439 (+1320675us) ts_tablet_manager.cc:1151] Deleting tablet 41bdb846ca19425b882434df0167479e
0924 19:57:41.257506 (+ 67us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.257722 (+ 216us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1320136}
W20260924 19:57:41.263636 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 41bdb846ca19425b882434df0167479e because the tablet was not found. No further retry: Not found: Tablet not found: 41bdb846ca19425b882434df0167479e
I20260924 19:57:41.265795 18224 tablet_bootstrap.cc:492] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:41.266333 18224 ts_tablet_manager.cc:1403] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.105s user 0.009s sys 0.000s
I20260924 19:57:41.269524 18224 raft_consensus.cc:359] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.270371 18224 raft_consensus.cc:385] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:41.270690 18224 raft_consensus.cc:740] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:41.271553 18224 consensus_queue.cc:260] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.274281 18224 ts_tablet_manager.cc:1434] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 19:57:41.254526 18220 tablet_bootstrap.cc:492] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:41.256708 18910 ts_tablet_manager.cc:1916] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:41.305425 18777 tablet_replica.cc:333] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:41.306257 18777 raft_consensus.cc:2243] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:41.307195 18777 raft_consensus.cc:2272] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:41.309948 18777 ts_tablet_manager.cc:1916] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:41.317028 18320 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/master-0/data/info.pb
I20260924 19:57:41.320430 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 18320
W20260924 19:57:41.333343 17601 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 78801b8d0b664217a74b20160ba2d8ac (table default.Table_454683a3d79d4988b41292f7776df549 [id=81b26242949442c0a7db6f55f0f18097], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 78801b8d0b664217a74b20160ba2d8ac consensus state has no leader
I20260924 19:57:41.349829 18223 tablet_bootstrap.cc:492] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:41.350257 18223 ts_tablet_manager.cc:1403] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.122s user 0.012s sys 0.036s
I20260924 19:57:41.352697 18223 raft_consensus.cc:359] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.353286 18223 raft_consensus.cc:385] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:41.353538 18223 raft_consensus.cc:740] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:41.354156 18223 consensus_queue.cc:260] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.357015 18220 tablet_bootstrap.cc:654] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:41.357301 18223 ts_tablet_manager.cc:1434] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 19:57:41.358171 18223 tablet_bootstrap.cc:492] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:41.363010 17753 consensus_peers.cc:597] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:41.370728 18777 ts_tablet_manager.cc:1929] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:41.371137 18777 log.cc:1199] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/dd2f4eb8bf254c4e85b6a9072271766b
I20260924 19:57:41.380367 18777 ts_tablet_manager.cc:1950] T dd2f4eb8bf254c4e85b6a9072271766b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:41.384847 18966 raft_consensus.cc:493] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:41.385309 18966 raft_consensus.cc:515] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.387838 18951 raft_consensus.cc:493] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:41.388320 18951 raft_consensus.cc:515] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.390247 18951 leader_election.cc:290] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:41.392544 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba46ac86b242407abd3e8dbf09f12efa" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:41.394893 17753 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:41.395876 18966 leader_election.cc:290] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:41.397637 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba46ac86b242407abd3e8dbf09f12efa" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:41.399214 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07f86f24d9d14d05898e1c0f69428850" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:41.400738 17753 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:41.401132 17753 leader_election.cc:304] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:41.403097 18970 raft_consensus.cc:2749] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:41.407349 18050 leader_election.cc:343] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:41.382283 18777 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 181) took 1246 ms. Trace:
I20260924 19:57:41.411064 18777 rpcz_store.cc:276] 0924 19:57:40.136100 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.142657 (+ 6557us) service_pool.cc:224] Handling call
0924 19:57:41.263756 (+1121099us) ts_tablet_manager.cc:1151] Deleting tablet dd2f4eb8bf254c4e85b6a9072271766b
0924 19:57:41.305271 (+ 41515us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.330125 (+ 24854us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.370583 (+ 40458us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.382221 (+ 11638us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1048229}
I20260924 19:57:41.412422 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet dd2f4eb8bf254c4e85b6a9072271766b (table default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b]) successfully deleted
I20260924 19:57:41.415289 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07f86f24d9d14d05898e1c0f69428850" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:41.415876 17816 raft_consensus.cc:2468] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:41.416450 18777 tablet_replica.cc:333] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:41.416978 18051 leader_election.cc:304] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:41.417271 18777 raft_consensus.cc:2243] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:41.418215 18777 raft_consensus.cc:2272] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:41.418251 18951 raft_consensus.cc:2804] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:41.418792 18951 raft_consensus.cc:493] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:41.419175 18951 raft_consensus.cc:3060] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:41.421227 18019 heartbeater.cc:344] Connected to a master server at 127.17.2.190:46309
I20260924 19:57:41.422955 17872 heartbeater.cc:344] Connected to a master server at 127.17.2.190:46309
I20260924 19:57:41.436307 18951 raft_consensus.cc:515] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.448289 18220 tablet_bootstrap.cc:492] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:41.448699 18220 ts_tablet_manager.cc:1403] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.194s user 0.005s sys 0.006s
I20260924 19:57:41.439507 18777 ts_tablet_manager.cc:1916] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:41.461290 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 167) took 3339 ms. Trace:
I20260924 19:57:41.461616 18231 rpcz_store.cc:276] 0924 19:57:38.122368 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:38.122503 (+ 135us) service_pool.cc:224] Handling call
0924 19:57:41.276752 (+3154249us) ts_tablet_manager.cc:1151] Deleting tablet 57008266b80345e0a3f8e60bd7e2d62b
0924 19:57:41.460984 (+184232us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.461231 (+ 247us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3153728}
I20260924 19:57:41.462585 18231 tablet_replica.cc:333] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:41.463303 18231 raft_consensus.cc:2243] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:41.464133 18231 raft_consensus.cc:2272] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:41.466333 18231 ts_tablet_manager.cc:1916] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:41.468245 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 57008266b80345e0a3f8e60bd7e2d62b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57008266b80345e0a3f8e60bd7e2d62b already in progress: creating tablet
I20260924 19:57:41.469619 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07f86f24d9d14d05898e1c0f69428850" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:41.470359 17816 raft_consensus.cc:3060] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:41.475878 18224 tablet_bootstrap.cc:492] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:41.489336 18166 heartbeater.cc:344] Connected to a master server at 127.17.2.190:46309
I20260924 19:57:41.492734 17816 raft_consensus.cc:2468] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:41.496078 18051 leader_election.cc:304] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:41.497318 18965 raft_consensus.cc:2804] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:41.497740 18965 raft_consensus.cc:697] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:41.498467 18965 consensus_queue.cc:237] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.504637 18951 leader_election.cc:290] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:41.504791 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07f86f24d9d14d05898e1c0f69428850" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
W20260924 19:57:41.506045 18050 leader_election.cc:343] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:41.507004 18220 raft_consensus.cc:359] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:41.507819 18220 raft_consensus.cc:385] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:41.508371 18777 ts_tablet_manager.cc:1929] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260924 19:57:41.508759 18777 log.cc:1199] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/83ee43b6dc064dc7b320691d52e57ea7
I20260924 19:57:41.509510 18777 ts_tablet_manager.cc:1950] T 83ee43b6dc064dc7b320691d52e57ea7 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:41.513197 18220 raft_consensus.cc:740] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:41.513983 18220 consensus_queue.cc:260] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:41.516058 19030 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:41.516803 19030 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:41.523525 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]) successfully deleted
I20260924 19:57:41.533002 18224 tablet_bootstrap.cc:654] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:41.544560 18223 tablet_bootstrap.cc:654] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:41.569911 18220 ts_tablet_manager.cc:1434] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.121s user 0.003s sys 0.002s
I20260924 19:57:41.570889 18220 tablet_bootstrap.cc:492] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:41.573063 18910 ts_tablet_manager.cc:1929] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:41.573508 18910 log.cc:1199] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/c884f12860164f12bbcd0d2f87067013
I20260924 19:57:41.574290 18910 ts_tablet_manager.cc:1950] T c884f12860164f12bbcd0d2f87067013 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:41.580297 19030 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:41.580718 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 253 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:41.598296 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 170) took 1632 ms. Trace:
I20260924 19:57:41.598668 18910 rpcz_store.cc:276] 0924 19:57:39.943244 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.979062 (+ 35818us) service_pool.cc:224] Handling call
0924 19:57:41.149852 (+1170790us) ts_tablet_manager.cc:1151] Deleting tablet c884f12860164f12bbcd0d2f87067013
0924 19:57:41.149906 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.341681 (+191775us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.572918 (+231237us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.575116 (+ 2198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8027,"tablet-delete.queue_time_us":1169064}
I20260924 19:57:41.600127 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet c884f12860164f12bbcd0d2f87067013 (table default.Table_5b3a0213ccbb4622a13a68e24d12af4e [id=54f05b78c3324978b2635be4bd969f66]) successfully deleted
I20260924 19:57:41.601545 18223 tablet_bootstrap.cc:492] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:41.601997 18223 ts_tablet_manager.cc:1403] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.244s user 0.007s sys 0.003s
I20260924 19:57:41.613178 18223 raft_consensus.cc:359] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.617604 19025 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:57:41.618693 18220 tablet_bootstrap.cc:654] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:41.619748 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 177) took 1674 ms. Trace:
I20260924 19:57:41.619956 18224 tablet_bootstrap.cc:492] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:41.620636 18223 raft_consensus.cc:385] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:41.620111 18910 rpcz_store.cc:276] 0924 19:57:39.945766 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.981643 (+ 35877us) service_pool.cc:224] Handling call
0924 19:57:41.619443 (+1637800us) ts_tablet_manager.cc:1151] Deleting tablet 24f84032bca24a7ca9da65f0bbceb0d5
0924 19:57:41.619493 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.619690 (+ 197us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1636860}
I20260924 19:57:41.623253 18223 raft_consensus.cc:740] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
W20260924 19:57:41.624250 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 24f84032bca24a7ca9da65f0bbceb0d5 because the tablet was not found. No further retry: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:41.624739 18224 ts_tablet_manager.cc:1403] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.149s user 0.010s sys 0.000s
I20260924 19:57:41.637815 18910 tablet_replica.cc:333] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:41.638746 18910 raft_consensus.cc:2243] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:41.639678 18910 raft_consensus.cc:2272] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:41.642081 18910 ts_tablet_manager.cc:1916] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:41.638418 18223 consensus_queue.cc:260] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.623060 18231 ts_tablet_manager.cc:1929] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:41.648169 18231 log.cc:1199] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/c884f12860164f12bbcd0d2f87067013
I20260924 19:57:41.664106 18223 ts_tablet_manager.cc:1434] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.062s user 0.005s sys 0.001s
I20260924 19:57:41.656329 18224 raft_consensus.cc:359] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.690333 18224 raft_consensus.cc:385] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:41.690728 18224 raft_consensus.cc:740] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:41.694416 17945 tablet_service.cc:1511] Processing CreateTablet for tablet d930e7f4a6b7471899dab1b28e40f93d (DEFAULT_TABLE table=default.Table_97a3c9fb681344628ba099ed831d60b3 [id=4fc8e565a6a640549716cee596fa3f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.695726 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d930e7f4a6b7471899dab1b28e40f93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.697695 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:41.698763 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:41.703884 18082 tablet_service.cc:1511] Processing CreateTablet for tablet d930e7f4a6b7471899dab1b28e40f93d (DEFAULT_TABLE table=default.Table_97a3c9fb681344628ba099ed831d60b3 [id=4fc8e565a6a640549716cee596fa3f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.705021 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d930e7f4a6b7471899dab1b28e40f93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.712239 17801 tablet_service.cc:1511] Processing CreateTablet for tablet d930e7f4a6b7471899dab1b28e40f93d (DEFAULT_TABLE table=default.Table_97a3c9fb681344628ba099ed831d60b3 [id=4fc8e565a6a640549716cee596fa3f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.712384 17949 tablet_service.cc:1511] Processing CreateTablet for tablet 96379e7356ab40218c6ee2c126c0b4ae (DEFAULT_TABLE table=default.Table_39ac82df7ce14073912622fcd1b57aaa [id=1ae05845cc1641ffaf3b82b0f1390e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.713215 17801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d930e7f4a6b7471899dab1b28e40f93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.713274 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96379e7356ab40218c6ee2c126c0b4ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.716269 18965 consensus_queue.cc:1048] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:41.726435 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 96379e7356ab40218c6ee2c126c0b4ae (DEFAULT_TABLE table=default.Table_39ac82df7ce14073912622fcd1b57aaa [id=1ae05845cc1641ffaf3b82b0f1390e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.727684 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96379e7356ab40218c6ee2c126c0b4ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.729517 18224 consensus_queue.cc:260] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.731525 18231 ts_tablet_manager.cc:1950] T c884f12860164f12bbcd0d2f87067013 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:41.747856 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]). Expected version 2 got 1
I20260924 19:57:41.749064 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_7b49c63ad79c4132afbbbb9e1f2d4ca0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 3 got 2
I20260924 19:57:41.751034 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
I20260924 19:57:41.755651 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 170) took 1973 ms. Trace:
I20260924 19:57:41.756013 18231 rpcz_store.cc:276] 0924 19:57:39.782375 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.782591 (+ 216us) service_pool.cc:224] Handling call
0924 19:57:41.462446 (+1679855us) ts_tablet_manager.cc:1151] Deleting tablet c884f12860164f12bbcd0d2f87067013
0924 19:57:41.462485 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.545424 (+ 82939us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.622909 (+ 77485us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:41.755584 (+132675us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1679286}
I20260924 19:57:41.757421 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet c884f12860164f12bbcd0d2f87067013 (table default.Table_5b3a0213ccbb4622a13a68e24d12af4e [id=54f05b78c3324978b2635be4bd969f66]) successfully deleted
I20260924 19:57:41.759588 18223 tablet_bootstrap.cc:492] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:41.765386 18223 tablet_bootstrap.cc:654] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:41.768784 18224 ts_tablet_manager.cc:1434] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.144s user 0.008s sys 0.001s
I20260924 19:57:41.781908 17796 tablet_service.cc:1511] Processing CreateTablet for tablet 96379e7356ab40218c6ee2c126c0b4ae (DEFAULT_TABLE table=default.Table_39ac82df7ce14073912622fcd1b57aaa [id=1ae05845cc1641ffaf3b82b0f1390e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.782967 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96379e7356ab40218c6ee2c126c0b4ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.782786 17947 tablet_service.cc:1511] Processing CreateTablet for tablet 6fd0d25961d74b67a42323a045f06b61 (DEFAULT_TABLE table=default.Table_fbc5ceeb4d1242d19344825cefb6ef2a [id=caf8f776b4754b80a6aac8211f1b7a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.783766 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd0d25961d74b67a42323a045f06b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.811607 18224 tablet_bootstrap.cc:492] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:41.813090 19025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Bootstrap starting.
I20260924 19:57:41.821338 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 173) took 1985 ms. Trace:
I20260924 19:57:41.825966 17609 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:41.835925 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 6fd0d25961d74b67a42323a045f06b61 (DEFAULT_TABLE table=default.Table_fbc5ceeb4d1242d19344825cefb6ef2a [id=caf8f776b4754b80a6aac8211f1b7a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.837206 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd0d25961d74b67a42323a045f06b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.863550 18220 tablet_bootstrap.cc:492] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:41.864059 18220 ts_tablet_manager.cc:1403] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.293s user 0.010s sys 0.004s
I20260924 19:57:41.866420 18220 raft_consensus.cc:359] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.875490 17799 tablet_service.cc:1511] Processing CreateTablet for tablet 5e9f9b76a58e47c0beee5ce94a44b707 (DEFAULT_TABLE table=default.Table_c449d7c2b02a4bcdb7091a3560bcaa02 [id=ff6c3885d44548b78bb3e4c556a5f383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.876499 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f9b76a58e47c0beee5ce94a44b707. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.827128 17609 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:41.878597 17609 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:41.787565 18965 raft_consensus.cc:493] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:41.879385 18965 raft_consensus.cc:515] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:41.879335 17609 catalog_manager.cc:5838] T c0feecd2fdb4409f86f6a5fbfbab3537 P c4f6e3a7b06148918d230c77ab0c8202 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:41.881276 17609 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_7b49c63ad79c4132afbbbb9e1f2d4ca0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 3 got 2
I20260924 19:57:41.881343 18965 leader_election.cc:290] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:41.833709 18084 tablet_service.cc:1511] Processing CreateTablet for tablet 6fd0d25961d74b67a42323a045f06b61 (DEFAULT_TABLE table=default.Table_fbc5ceeb4d1242d19344825cefb6ef2a [id=caf8f776b4754b80a6aac8211f1b7a6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.882164 17609 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]). Expected version 2 got 1
I20260924 19:57:41.882443 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd0d25961d74b67a42323a045f06b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.884117 17609 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
I20260924 19:57:41.884873 17609 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:41.873698 18951 raft_consensus.cc:493] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:41.889761 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affce4de8e0b412aa3b4bf19f9be7b6e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:41.890332 17821 raft_consensus.cc:2468] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:41.890080 18951 raft_consensus.cc:515] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.891991 18951 leader_election.cc:290] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:41.894065 18051 leader_election.cc:304] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:41.897568 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0246d83189d4ba5b75fbd8390f517f6" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:41.898068 17821 raft_consensus.cc:2468] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:41.899312 18051 leader_election.cc:304] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:41.870779 17940 tablet_service.cc:1511] Processing CreateTablet for tablet 5e9f9b76a58e47c0beee5ce94a44b707 (DEFAULT_TABLE table=default.Table_c449d7c2b02a4bcdb7091a3560bcaa02 [id=ff6c3885d44548b78bb3e4c556a5f383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.899811 18220 raft_consensus.cc:385] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:41.900120 18220 raft_consensus.cc:740] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:41.900521 17940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f9b76a58e47c0beee5ce94a44b707. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.901618 18965 raft_consensus.cc:2804] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:41.901940 18965 raft_consensus.cc:493] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:41.902252 18965 raft_consensus.cc:3060] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:41.904361 17797 tablet_service.cc:1511] Processing CreateTablet for tablet b25dd98712d54cd5b53df00b2cf9f3c4 (DEFAULT_TABLE table=default.Table_e91b551877474bb0914b854a58447b3a [id=0606053a508145868b640f364fbf48d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.905427 17797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25dd98712d54cd5b53df00b2cf9f3c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.906018 17947 tablet_service.cc:1511] Processing CreateTablet for tablet cfe0303596204e528585c4fe1c476d8b (DEFAULT_TABLE table=default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.906968 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfe0303596204e528585c4fe1c476d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.875104 17950 tablet_service.cc:1511] Processing CreateTablet for tablet b25dd98712d54cd5b53df00b2cf9f3c4 (DEFAULT_TABLE table=default.Table_e91b551877474bb0914b854a58447b3a [id=0606053a508145868b640f364fbf48d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.907596 18963 raft_consensus.cc:2804] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:41.907779 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25dd98712d54cd5b53df00b2cf9f3c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.907938 18963 raft_consensus.cc:493] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:41.908263 18963 raft_consensus.cc:3060] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:41.908692 18082 tablet_service.cc:1511] Processing CreateTablet for tablet b25dd98712d54cd5b53df00b2cf9f3c4 (DEFAULT_TABLE table=default.Table_e91b551877474bb0914b854a58447b3a [id=0606053a508145868b640f364fbf48d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.909740 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25dd98712d54cd5b53df00b2cf9f3c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.899395 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 5e9f9b76a58e47c0beee5ce94a44b707 (DEFAULT_TABLE table=default.Table_c449d7c2b02a4bcdb7091a3560bcaa02 [id=ff6c3885d44548b78bb3e4c556a5f383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.912375 18220 consensus_queue.cc:260] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.914515 18220 ts_tablet_manager.cc:1434] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.050s user 0.003s sys 0.001s
I20260924 19:57:41.895896 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affce4de8e0b412aa3b4bf19f9be7b6e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:41.896592 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0246d83189d4ba5b75fbd8390f517f6" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:41.916823 18780 raft_consensus.cc:493] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:41.917281 18780 raft_consensus.cc:515] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.920430 18780 leader_election.cc:290] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:41.921159 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f9b76a58e47c0beee5ce94a44b707. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.921150 17795 tablet_service.cc:1511] Processing CreateTablet for tablet cfe0303596204e528585c4fe1c476d8b (DEFAULT_TABLE table=default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.922190 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfe0303596204e528585c4fe1c476d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.922220 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44199d2655f14dcf97afe2180789aae4" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:41.924157 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44199d2655f14dcf97afe2180789aae4" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:41.925530 17753 leader_election.cc:343] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:41.926126 17753 leader_election.cc:343] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:41.926461 17753 leader_election.cc:304] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:41.931404 18780 raft_consensus.cc:2749] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:41.932942 18095 tablet_service.cc:1511] Processing CreateTablet for tablet cfe0303596204e528585c4fe1c476d8b (DEFAULT_TABLE table=default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.934177 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfe0303596204e528585c4fe1c476d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.951192 17940 tablet_service.cc:1511] Processing CreateTablet for tablet 335f979eb04c49bcb53023693f54ad41 (DEFAULT_TABLE table=default.Table_ae1050d7716043fbad49a8f62505e719 [id=8681c352f02242a69511794ee585bf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.953732 17940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 335f979eb04c49bcb53023693f54ad41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.958076 18090 tablet_service.cc:1511] Processing CreateTablet for tablet 335f979eb04c49bcb53023693f54ad41 (DEFAULT_TABLE table=default.Table_ae1050d7716043fbad49a8f62505e719 [id=8681c352f02242a69511794ee585bf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:41.959194 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 335f979eb04c49bcb53023693f54ad41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:41.974030 18220 tablet_bootstrap.cc:492] T 3b328d37d55e4116a29322caaa21d431 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:41.982605 18963 raft_consensus.cc:515] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:41.984598 18963 leader_election.cc:290] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:42.012795 18224 tablet_bootstrap.cc:654] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:42.026423 18909 raft_consensus.cc:493] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:42.027769 18909 raft_consensus.cc:515] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.031782 18909 leader_election.cc:290] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:42.035655 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 137158us
W20260924 19:57:42.036525 18050 leader_election.cc:343] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:42.037467 18050 leader_election.cc:343] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:42.039362 17970 raft_consensus.cc:3060] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:42.059722 17610 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet f90a677e94034c73ab5452554862c1c3
I20260924 19:57:42.061215 17610 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 213) took 1991 ms. Trace:
I20260924 19:57:42.061575 17610 rpcz_store.cc:276] 0924 19:57:40.069691 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.445921 (+376230us) service_pool.cc:224] Handling call
0924 19:57:40.482751 (+ 36830us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:40.575476 (+ 92725us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:40.575603 (+ 127us) catalog_manager.cc:2863] Locking tablets
0924 19:57:40.911788 (+336185us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:40.938515 (+ 26727us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:40.938790 (+ 275us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:42.059635 (+1120845us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:42.061148 (+ 1513us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:40.915685 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:40.916163 (+ 478us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:40.916190 (+ 27us) write_op.cc:435] Acquired schema lock
0924 19:57:40.916208 (+ 18us) tablet.cc:662] Decoding operations
0924 19:57:40.916549 (+ 341us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:40.916586 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:40.916613 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:40.916926 (+ 313us) tablet.cc:701] Row locks acquired
0924 19:57:40.916942 (+ 16us) write_op.cc:260] PREPARE: finished
0924 19:57:40.917069 (+ 127us) write_op.cc:270] Start()
0924 19:57:40.917157 (+ 88us) write_op.cc:276] Timestamp: P: 1790279860917028 usec, L: 0
0924 19:57:40.917174 (+ 17us) op_driver.cc:348] REPLICATION: starting
0924 19:57:40.917707 (+ 533us) log.cc:844] Serialized 762 byte log entry
0924 19:57:40.936810 (+ 19103us) op_driver.cc:464] REPLICATION: finished
0924 19:57:40.937142 (+ 332us) write_op.cc:301] APPLY: starting
0924 19:57:40.937196 (+ 54us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:40.937299 (+ 103us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:40.937396 (+ 97us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:40.937685 (+ 289us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:40.937822 (+ 137us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:40.937862 (+ 40us) write_op.cc:312] APPLY: finished
0924 19:57:40.938049 (+ 187us) log.cc:844] Serialized 29 byte log entry
0924 19:57:40.938212 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.938288 (+ 76us) write_op.cc:454] Released schema lock
0924 19:57:40.938323 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:40.938546 (+ 223us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.938568 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":186,"apply.run_cpu_time_us":1374,"apply.run_wall_time_us":1373,"num_ops":2,"prepare.queue_time_us":2166,"prepare.run_cpu_time_us":2237,"prepare.run_wall_time_us":2237,"replication_time_us":19520}]]}
I20260924 19:57:42.071421 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet f90a677e94034c73ab5452554862c1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:40 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:42.076839 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0246d83189d4ba5b75fbd8390f517f6" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:42.081959 18220 tablet_bootstrap.cc:654] T 3b328d37d55e4116a29322caaa21d431 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:42.086123 18050 consensus_peers.cc:597] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260924 19:57:42.086717 18050 leader_election.cc:343] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:42.091378 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet f90a677e94034c73ab5452554862c1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:40 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:42.090626 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c08f10d2a364cb8a0a7ea40654c3fe4" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:42.106432 17606 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]). Expected version 2 got 1
I20260924 19:57:41.861279 18231 rpcz_store.cc:276] 0924 19:57:39.835674 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.835793 (+ 119us) service_pool.cc:224] Handling call
0924 19:57:41.821030 (+1985237us) ts_tablet_manager.cc:1151] Deleting tablet 5f994e0401df4fc19bc5abb10f08bc6e
0924 19:57:41.821080 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:41.821274 (+ 194us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1984714}
I20260924 19:57:42.032510 18780 raft_consensus.cc:493] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:42.108271 17606 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:42.108227 18780 raft_consensus.cc:515] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:42.108996 17606 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_7b49c63ad79c4132afbbbb9e1f2d4ca0 [id=8edea4510fd84d40819b48b401ccfe64]). Expected version 3 got 2
I20260924 19:57:42.110224 18780 leader_election.cc:290] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:42.109848 17606 catalog_manager.cc:5838] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:42.111130 17606 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:42.111796 17606 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:42.111707 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f169c47d711d4566b2004cdfd0de725d" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:42.112226 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 5f994e0401df4fc19bc5abb10f08bc6e because the tablet was not found. No further retry: Not found: Tablet not found: 5f994e0401df4fc19bc5abb10f08bc6e
I20260924 19:57:42.113204 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 174) took 2264 ms. Trace:
W20260924 19:57:42.113838 17753 leader_election.cc:343] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:42.113576 17606 catalog_manager.cc:5838] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:42.113746 18231 rpcz_store.cc:276] 0924 19:57:39.848997 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.849153 (+ 156us) service_pool.cc:224] Handling call
0924 19:57:42.112957 (+2263804us) ts_tablet_manager.cc:1151] Deleting tablet fec9b6ecfb384b528cf71c4cfb2fd4cb
0924 19:57:42.112998 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:42.113149 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2263113}
I20260924 19:57:42.114740 17606 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
W20260924 19:57:42.115146 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb because the tablet was not found. No further retry: Not found: Tablet not found: fec9b6ecfb384b528cf71c4cfb2fd4cb
I20260924 19:57:42.116107 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 175) took 2261 ms. Trace:
I20260924 19:57:42.070614 17870 heartbeater.cc:344] Connected to a master server at 127.17.2.188:38761
I20260924 19:57:42.115727 17606 catalog_manager.cc:5838] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:42.116859 17606 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:42.117539 17613 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 910d61b8c3574302b5a328187226571e
I20260924 19:57:42.119037 17613 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 212) took 2126 ms. Trace:
I20260924 19:57:42.121588 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f169c47d711d4566b2004cdfd0de725d" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:42.122958 18092 tablet_service.cc:1558] Processing DeleteTablet for tablet 910d61b8c3574302b5a328187226571e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:40 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:42.124675 17615 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe] alter complete (version 1)
W20260924 19:57:42.126164 17753 leader_election.cc:343] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:42.126552 17753 leader_election.cc:304] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:42.129717 18223 tablet_bootstrap.cc:492] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:42.130136 18223 ts_tablet_manager.cc:1403] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.371s user 0.012s sys 0.009s
I20260924 19:57:42.130684 18780 raft_consensus.cc:2749] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:42.133250 18965 raft_consensus.cc:515] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:42.119343 17613 rpcz_store.cc:276] 0924 19:57:39.992581 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.428354 (+435773us) service_pool.cc:224] Handling call
0924 19:57:40.429842 (+ 1488us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:40.571422 (+141580us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:40.571583 (+ 161us) catalog_manager.cc:2863] Locking tablets
0924 19:57:40.900476 (+328893us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:40.922960 (+ 22484us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:40.923591 (+ 631us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:42.117458 (+1193867us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:42.118979 (+ 1521us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:40.902435 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:40.902908 (+ 473us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:40.902936 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:57:40.902953 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:40.903343 (+ 390us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:40.903389 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:40.903420 (+ 31us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:40.903741 (+ 321us) tablet.cc:701] Row locks acquired
0924 19:57:40.903755 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:40.903906 (+ 151us) write_op.cc:270] Start()
0924 19:57:40.904003 (+ 97us) write_op.cc:276] Timestamp: P: 1790279860903849 usec, L: 0
0924 19:57:40.904020 (+ 17us) op_driver.cc:348] REPLICATION: starting
0924 19:57:40.904568 (+ 548us) log.cc:844] Serialized 762 byte log entry
0924 19:57:40.919342 (+ 14774us) op_driver.cc:464] REPLICATION: finished
0924 19:57:40.921320 (+ 1978us) write_op.cc:301] APPLY: starting
0924 19:57:40.921378 (+ 58us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:40.921488 (+ 110us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:40.921576 (+ 88us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:40.921836 (+ 260us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:40.921982 (+ 146us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:40.922038 (+ 56us) write_op.cc:312] APPLY: finished
0924 19:57:40.922213 (+ 175us) log.cc:844] Serialized 29 byte log entry
0924 19:57:40.922351 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.922404 (+ 53us) write_op.cc:454] Released schema lock
0924 19:57:40.922436 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:40.922643 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.922669 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1856,"apply.run_cpu_time_us":1284,"apply.run_wall_time_us":1284,"num_ops":2,"prepare.queue_time_us":198,"prepare.run_cpu_time_us":2489,"prepare.run_wall_time_us":4900,"raft.queue_time_us":1066,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":15249}]]}
I20260924 19:57:42.078212 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet f90a677e94034c73ab5452554862c1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:40 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:42.137044 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet c884f12860164f12bbcd0d2f87067013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:38 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:42.075872 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 335f979eb04c49bcb53023693f54ad41 (DEFAULT_TABLE table=default.Table_ae1050d7716043fbad49a8f62505e719 [id=8681c352f02242a69511794ee585bf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:42.104105 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0246d83189d4ba5b75fbd8390f517f6" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:42.138582 17821 raft_consensus.cc:3060] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:42.138672 17613 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_1dad27c8fd4445fb89e32722eef0abe2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:42.140342 18017 heartbeater.cc:344] Connected to a master server at 127.17.2.188:38761
I20260924 19:57:42.143987 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 335f979eb04c49bcb53023693f54ad41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:42.058873 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c08f10d2a364cb8a0a7ea40654c3fe4" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:42.145390 17906 leader_election.cc:304] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
W20260924 19:57:42.147063 17522 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.190:46309 timed out after 0.627s (SENT)
I20260924 19:57:42.147996 17821 raft_consensus.cc:2468] T e0246d83189d4ba5b75fbd8390f517f6 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:42.149070 19046 raft_consensus.cc:2749] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:42.149511 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affce4de8e0b412aa3b4bf19f9be7b6e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:42.149739 18231 rpcz_store.cc:276] 0924 19:57:39.854845 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.854971 (+ 126us) service_pool.cc:224] Handling call
0924 19:57:42.115887 (+2260916us) ts_tablet_manager.cc:1151] Deleting tablet 349c6e9413bb41268d0c41d549203e2a
0924 19:57:42.115924 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:42.116060 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2260379}
W20260924 19:57:42.151340 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 349c6e9413bb41268d0c41d549203e2a because the tablet was not found. No further retry: Not found: Tablet not found: 349c6e9413bb41268d0c41d549203e2a
I20260924 19:57:42.152539 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 176) took 2295 ms. Trace:
I20260924 19:57:42.152878 18231 rpcz_store.cc:276] 0924 19:57:39.856854 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.856971 (+ 117us) service_pool.cc:224] Handling call
0924 19:57:42.152279 (+2295308us) ts_tablet_manager.cc:1151] Deleting tablet 24f84032bca24a7ca9da65f0bbceb0d5
0924 19:57:42.152322 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:42.152487 (+ 165us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2294812}
I20260924 19:57:42.154170 17950 tablet_service.cc:1558] Processing DeleteTablet for tablet a765abd702644f079bddffb3fc4238ef with delete_type TABLET_DATA_DELETED (replaced by tablet f169c47d711d4566b2004cdfd0de725d) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:42.154603 17615 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 38) took 3392 ms (client timeout 1000 ms). Trace:
I20260924 19:57:42.154829 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:42.155441 17950 tablet_service.cc:1558] Processing DeleteTablet for tablet 41bdb846ca19425b882434df0167479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:31 UTC) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:42.155862 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 24f84032bca24a7ca9da65f0bbceb0d5 because the tablet was not found. No further retry: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5
W20260924 19:57:42.146332 17613 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dad27c8fd4445fb89e32722eef0abe2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:42.159453 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 2cf62a73c49245198d65011875148dc3 with delete_type TABLET_DATA_DELETED (replaced by tablet cfe0303596204e528585c4fe1c476d8b) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:42.154995 17615 rpcz_store.cc:269] 0924 19:57:38.762141 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:38.836808 (+ 74667us) service_pool.cc:224] Handling call
0924 19:57:38.898236 (+ 61428us) spinlock_profiling.cc:243] Waited 57.1 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:39.927728 (+1029492us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:39.927764 (+ 36us) write_op.cc:454] Released schema lock
0924 19:57:42.154522 (+2226758us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:39.877297 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:39.877783 (+ 486us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:39.877810 (+ 27us) write_op.cc:435] Acquired schema lock
0924 19:57:39.877831 (+ 21us) tablet.cc:662] Decoding operations
0924 19:57:39.878138 (+ 307us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:39.878185 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:39.878213 (+ 28us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:39.878379 (+ 166us) tablet.cc:701] Row locks acquired
0924 19:57:39.878398 (+ 19us) write_op.cc:260] PREPARE: finished
0924 19:57:39.878531 (+ 133us) write_op.cc:270] Start()
0924 19:57:39.878627 (+ 96us) write_op.cc:276] Timestamp: P: 1790279859878483 usec, L: 0
0924 19:57:39.878646 (+ 19us) op_driver.cc:348] REPLICATION: starting
0924 19:57:39.879233 (+ 587us) log.cc:844] Serialized 356 byte log entry
0924 19:57:39.894514 (+ 15281us) op_driver.cc:464] REPLICATION: finished
0924 19:57:39.895114 (+ 600us) write_op.cc:301] APPLY: starting
0924 19:57:39.895186 (+ 72us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:39.895290 (+ 104us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:39.895386 (+ 96us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:39.895553 (+ 167us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:39.895685 (+ 132us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:39.895726 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:57:39.895904 (+ 178us) log.cc:844] Serialized 23 byte log entry
0924 19:57:39.896068 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:39.896127 (+ 59us) write_op.cc:454] Released schema lock
0924 19:57:39.896158 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":20804,"reported_tablets":22,"spinlock_wait_cycles":131253504,"child_traces":[["op",{"apply.queue_time_us":459,"apply.run_cpu_time_us":1215,"apply.run_wall_time_us":1215,"num_ops":1,"prepare.queue_time_us":117,"prepare.run_cpu_time_us":2318,"prepare.run_wall_time_us":2320,"raft.queue_time_us":6195,"raft.run_cpu_time_us":919,"raft.run_wall_time_us":920,"replication_time_us":15774}]]}
I20260924 19:57:42.164489 17945 tablet_service.cc:1558] Processing DeleteTablet for tablet 910d61b8c3574302b5a328187226571e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:40 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:42.167284 18231 tablet_replica.cc:333] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:42.168123 18231 raft_consensus.cc:2243] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:42.168695 18231 raft_consensus.cc:2272] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:42.170838 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 24f84032bca24a7ca9da65f0bbceb0d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:30 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:42.181761 18231 ts_tablet_manager.cc:1916] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:42.190295 18778 raft_consensus.cc:493] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:42.190732 18778 raft_consensus.cc:515] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } }
I20260924 19:57:42.193279 18778 leader_election.cc:290] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:42.202143 17607 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 29) took 2796 ms (client timeout 1000 ms). Trace:
W20260924 19:57:42.203894 17906 leader_election.cc:343] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: fec9b6ecfb384b528cf71c4cfb2fd4cb
W20260924 19:57:42.215788 17905 leader_election.cc:343] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: fec9b6ecfb384b528cf71c4cfb2fd4cb
I20260924 19:57:42.216207 17905 leader_election.cc:304] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:42.219272 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet d970cc92a1134a98a613d198a319cb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:42.220084 19046 raft_consensus.cc:2749] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:42.135306 18965 leader_election.cc:290] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
W20260924 19:57:42.150717 18050 leader_election.cc:343] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:42.227416 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 76728us
I20260924 19:57:42.182778 18092 tablet_service.cc:1558] Processing DeleteTablet for tablet 151247d6b5254cd884c58db9311ac04a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:42.184881 18087 tablet_service.cc:1558] Processing DeleteTablet for tablet c884f12860164f12bbcd0d2f87067013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:38 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:42.185361 18089 tablet_service.cc:1558] Processing DeleteTablet for tablet 83ee43b6dc064dc7b320691d52e57ea7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c1afbfc7de7480ab4be40680a3e3ada) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:42.210451 18098 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a524bad1c4040e58ad84fcf5a4fc47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:37 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:42.212029 18085 tablet_service.cc:1558] Processing DeleteTablet for tablet d970cc92a1134a98a613d198a319cb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:42.212858 18097 tablet_service.cc:1558] Processing DeleteTablet for tablet 2cf62a73c49245198d65011875148dc3 with delete_type TABLET_DATA_DELETED (replaced by tablet cfe0303596204e528585c4fe1c476d8b) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:42.213665 18088 tablet_service.cc:1558] Processing DeleteTablet for tablet dd2f4eb8bf254c4e85b6a9072271766b with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1ac7e1b192465696640c02898bc39f) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:42.214520 18091 tablet_service.cc:1558] Processing DeleteTablet for tablet a765abd702644f079bddffb3fc4238ef with delete_type TABLET_DATA_DELETED (replaced by tablet f169c47d711d4566b2004cdfd0de725d) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:42.231873 19051 tablet_replica.cc:333] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
W20260924 19:57:42.232239 17614 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 41) took 2853 ms (client timeout 1000 ms). Trace:
I20260924 19:57:42.233069 19051 raft_consensus.cc:2243] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:42.234076 19051 raft_consensus.cc:2272] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:42.220577 17607 rpcz_store.cc:269] 0924 19:57:39.406162 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.787608 (+381446us) service_pool.cc:224] Handling call
0924 19:57:42.202074 (+2414466us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":25}
I20260924 19:57:42.238054 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_9716bbadd5a3442382dd3562939de32d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:42.217916 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 113585us
W20260924 19:57:42.239804 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9716bbadd5a3442382dd3562939de32d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:42.187588 17615 master_service.cc:960] ReplaceTablet: received request to replace tablet 57008266b80345e0a3f8e60bd7e2d62b from {username='slave'} at 127.0.0.1:51280
W20260924 19:57:42.211362 17608 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 39) took 1344 ms (client timeout 1000 ms). Trace:
W20260924 19:57:42.240613 17608 rpcz_store.cc:269] 0924 19:57:40.867239 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.936774 (+ 69535us) service_pool.cc:224] Handling call
0924 19:57:42.211287 (+1274513us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":24}
I20260924 19:57:42.224608 18963 raft_consensus.cc:493] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:42.170838 17796 tablet_service.cc:1558] Processing DeleteTablet for tablet 349c6e9413bb41268d0c41d549203e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:42.242110 18963 raft_consensus.cc:515] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:42.205914 17611 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 40) took 4205 ms (client timeout 1000 ms). Trace:
W20260924 19:57:42.243124 17611 rpcz_store.cc:269] 0924 19:57:38.000843 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:38.133382 (+132539us) service_pool.cc:224] Handling call
0924 19:57:42.205834 (+4072452us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:39.879743 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:39.880228 (+ 485us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:39.880258 (+ 30us) write_op.cc:435] Acquired schema lock
0924 19:57:39.880271 (+ 13us) tablet.cc:662] Decoding operations
0924 19:57:39.880566 (+ 295us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:39.880613 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:39.880645 (+ 32us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:39.880817 (+ 172us) tablet.cc:701] Row locks acquired
0924 19:57:39.880836 (+ 19us) write_op.cc:260] PREPARE: finished
0924 19:57:39.880970 (+ 134us) write_op.cc:270] Start()
0924 19:57:39.881065 (+ 95us) write_op.cc:276] Timestamp: P: 1790279859880924 usec, L: 0
0924 19:57:39.881094 (+ 29us) op_driver.cc:348] REPLICATION: starting
0924 19:57:39.881638 (+ 544us) log.cc:844] Serialized 530 byte log entry
0924 19:57:39.894715 (+ 13077us) op_driver.cc:464] REPLICATION: finished
0924 19:57:39.896430 (+ 1715us) write_op.cc:301] APPLY: starting
0924 19:57:39.896484 (+ 54us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:39.896575 (+ 91us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:39.896678 (+ 103us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:39.896814 (+ 136us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:39.896953 (+ 139us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:39.896998 (+ 45us) write_op.cc:312] APPLY: finished
0924 19:57:39.897151 (+ 153us) log.cc:844] Serialized 23 byte log entry
0924 19:57:39.897284 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:39.897343 (+ 59us) write_op.cc:454] Released schema lock
0924 19:57:39.897370 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":26320,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":1609,"apply.run_cpu_time_us":1106,"apply.run_wall_time_us":1107,"num_ops":1,"prepare.queue_time_us":2324,"prepare.run_cpu_time_us":2341,"prepare.run_wall_time_us":10412,"raft.queue_time_us":1551,"raft.run_cpu_time_us":23,"raft.run_wall_time_us":23,"replication_time_us":13556,"thread_start_us":123556,"threads_started":1}]]}
I20260924 19:57:42.247612 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_66edcc97f7a64dfa85bf24d26e49c5de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:42.249205 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66edcc97f7a64dfa85bf24d26e49c5de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:42.189934 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet 910d61b8c3574302b5a328187226571e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:40 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:42.194254 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 41bdb846ca19425b882434df0167479e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:31 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:42.195513 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet c884f12860164f12bbcd0d2f87067013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:38 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:42.204610 17802 tablet_service.cc:1558] Processing DeleteTablet for tablet 2cf62a73c49245198d65011875148dc3 with delete_type TABLET_DATA_DELETED (replaced by tablet cfe0303596204e528585c4fe1c476d8b) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:42.252009 18963 leader_election.cc:290] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:42.254274 17609 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 42) took 1638 ms (client timeout 1000 ms). Trace:
I20260924 19:57:42.256139 18220 tablet_bootstrap.cc:492] T 3b328d37d55e4116a29322caaa21d431 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:42.256538 18220 ts_tablet_manager.cc:1403] T 3b328d37d55e4116a29322caaa21d431 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.283s user 0.008s sys 0.002s
I20260924 19:57:42.261790 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet 07f86f24d9d14d05898e1c0f69428850 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d95d6e48173436c8c59130464413ea1) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:42.265130 18224 tablet_bootstrap.cc:492] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:42.265576 18224 ts_tablet_manager.cc:1403] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.458s user 0.010s sys 0.000s
I20260924 19:57:42.268283 18224 raft_consensus.cc:359] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.268911 18224 raft_consensus.cc:385] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:42.269168 18224 raft_consensus.cc:740] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
W20260924 19:57:42.268689 17609 rpcz_store.cc:269] 0924 19:57:40.616350 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.619946 (+ 3596us) service_pool.cc:224] Handling call
0924 19:57:42.254199 (+1634253us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:42.061608 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:42.062129 (+ 521us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:42.062160 (+ 31us) write_op.cc:435] Acquired schema lock
0924 19:57:42.062181 (+ 21us) tablet.cc:662] Decoding operations
0924 19:57:42.062504 (+ 323us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:42.062546 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:42.062573 (+ 27us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:42.062734 (+ 161us) tablet.cc:701] Row locks acquired
0924 19:57:42.062750 (+ 16us) write_op.cc:260] PREPARE: finished
0924 19:57:42.062896 (+ 146us) write_op.cc:270] Start()
0924 19:57:42.062991 (+ 95us) write_op.cc:276] Timestamp: P: 1790279862062814 usec, L: 0
0924 19:57:42.063009 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:42.063584 (+ 575us) log.cc:844] Serialized 530 byte log entry
0924 19:57:42.073379 (+ 9795us) op_driver.cc:464] REPLICATION: finished
0924 19:57:42.073613 (+ 234us) write_op.cc:301] APPLY: starting
0924 19:57:42.073676 (+ 63us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:42.073766 (+ 90us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:42.073852 (+ 86us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:42.074019 (+ 167us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:42.074149 (+ 130us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:42.074190 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:57:42.074356 (+ 166us) log.cc:844] Serialized 23 byte log entry
0924 19:57:42.074517 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.074576 (+ 59us) write_op.cc:454] Released schema lock
0924 19:57:42.074606 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:42.074808 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.074833 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":36856,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":82,"apply.run_cpu_time_us":1169,"apply.run_wall_time_us":1170,"num_ops":1,"prepare.queue_time_us":506,"prepare.run_cpu_time_us":2252,"prepare.run_wall_time_us":2253,"raft.queue_time_us":1178,"raft.run_cpu_time_us":659,"raft.run_wall_time_us":655,"replication_time_us":10279}]]}
I20260924 19:57:42.241405 18051 leader_election.cc:304] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:42.278651 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df3dcf4ddcf4efa9aa047da56dfb7a7" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:42.278551 17609 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_2255ca45bfcf4f819376770df3186e24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:42.279915 18050 leader_election.cc:343] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:42.283224 17609 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2255ca45bfcf4f819376770df3186e24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:42.284783 18951 raft_consensus.cc:2804] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:42.285218 18951 raft_consensus.cc:697] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:42.285985 18951 consensus_queue.cc:237] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.291491 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "affce4de8e0b412aa3b4bf19f9be7b6e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:42.292141 17821 raft_consensus.cc:3060] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:42.298089 17967 raft_consensus.cc:3060] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:42.232601 17614 rpcz_store.cc:269] 0924 19:57:39.378548 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.768720 (+390172us) service_pool.cc:224] Handling call
0924 19:57:42.232161 (+2463441us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:40.876101 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:40.876588 (+ 487us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:40.876623 (+ 35us) write_op.cc:435] Acquired schema lock
0924 19:57:40.876639 (+ 16us) tablet.cc:662] Decoding operations
0924 19:57:40.876971 (+ 332us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:40.877016 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:40.877045 (+ 29us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:40.877222 (+ 177us) tablet.cc:701] Row locks acquired
0924 19:57:40.877235 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:57:40.877369 (+ 134us) write_op.cc:270] Start()
0924 19:57:40.877467 (+ 98us) write_op.cc:276] Timestamp: P: 1790279860877332 usec, L: 0
0924 19:57:40.877482 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:57:40.878038 (+ 556us) log.cc:844] Serialized 530 byte log entry
0924 19:57:40.888938 (+ 10900us) op_driver.cc:464] REPLICATION: finished
0924 19:57:40.889309 (+ 371us) write_op.cc:301] APPLY: starting
0924 19:57:40.889395 (+ 86us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:40.889518 (+ 123us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:40.889645 (+ 127us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:40.889846 (+ 201us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:40.890014 (+ 168us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:40.890080 (+ 66us) write_op.cc:312] APPLY: finished
0924 19:57:40.890297 (+ 217us) log.cc:844] Serialized 23 byte log entry
0924 19:57:40.890492 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.890582 (+ 90us) write_op.cc:454] Released schema lock
0924 19:57:40.890622 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:40.890858 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.890886 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":56781,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":200,"apply.run_cpu_time_us":1519,"apply.run_wall_time_us":1519,"num_ops":1,"prepare.queue_time_us":38600,"prepare.run_cpu_time_us":2252,"prepare.run_wall_time_us":2252,"raft.queue_time_us":827,"raft.run_cpu_time_us":629,"raft.run_wall_time_us":629,"replication_time_us":11331}]]}
I20260924 19:57:42.312191 17821 raft_consensus.cc:2468] T affce4de8e0b412aa3b4bf19f9be7b6e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:42.312336 18910 ts_tablet_manager.cc:1929] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:42.313027 18224 consensus_queue.cc:260] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.336226 18224 ts_tablet_manager.cc:1434] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.070s user 0.006s sys 0.001s
I20260924 19:57:42.336833 18910 log.cc:1199] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/8a524bad1c4040e58ad84fcf5a4fc47d
I20260924 19:57:42.343659 18910 ts_tablet_manager.cc:1950] T 8a524bad1c4040e58ad84fcf5a4fc47d P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:42.348242 18224 tablet_bootstrap.cc:492] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:42.348807 19025 log.cc:826] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:57:42.258882 18220 raft_consensus.cc:359] T 3b328d37d55e4116a29322caaa21d431 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.356034 18220 raft_consensus.cc:385] T 3b328d37d55e4116a29322caaa21d431 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:42.356402 18220 raft_consensus.cc:740] T 3b328d37d55e4116a29322caaa21d431 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:42.357275 18220 consensus_queue.cc:260] T 3b328d37d55e4116a29322caaa21d431 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.359754 18220 ts_tablet_manager.cc:1434] T 3b328d37d55e4116a29322caaa21d431 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.103s user 0.007s sys 0.000s
I20260924 19:57:42.374104 17614 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_fe4242bbbc1f4747bd1f2b4d25c8bacd" } new_table_name: "default.Table_4e3a3158cde24302806b7ba7827831c4" modify_external_catalogs: true
I20260924 19:57:42.377115 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 174) took 2432 ms. Trace:
I20260924 19:57:42.385226 18220 tablet_bootstrap.cc:492] T 6bd442f16492495f84bb8363b9bade97 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:42.377609 18910 rpcz_store.cc:276] 0924 19:57:39.944757 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.982791 (+ 38034us) service_pool.cc:224] Handling call
0924 19:57:41.637654 (+1654863us) ts_tablet_manager.cc:1151] Deleting tablet 8a524bad1c4040e58ad84fcf5a4fc47d
0924 19:57:41.637696 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:42.105357 (+467661us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:42.312191 (+206834us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:42.377035 (+ 64844us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1653994}
I20260924 19:57:42.396432 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 176) took 2451 ms. Trace:
I20260924 19:57:42.396970 17612 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 211) took 2426 ms. Trace:
I20260924 19:57:42.397500 17615 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51280 (request call id 217) took 1715 ms. Trace:
I20260924 19:57:42.403352 17615 rpcz_store.cc:276] 0924 19:57:40.682420 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.186312 (+1503892us) service_pool.cc:224] Handling call
0924 19:57:42.397259 (+210947us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:42.405506 17615 master_service.cc:960] ReplaceTablet: received request to replace tablet ae7a0660795e4e4b87d0959ca84d5d1b from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:42.399667 17612 rpcz_store.cc:276] 0924 19:57:39.970962 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.130582 (+159620us) service_pool.cc:224] Handling call
0924 19:57:40.154579 (+ 23997us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:40.156678 (+ 2099us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:40.158773 (+ 2095us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:40.158996 (+ 223us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:40.569350 (+410354us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:42.396717 (+1827367us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:40.160680 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:40.162242 (+ 1562us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:40.162279 (+ 37us) write_op.cc:435] Acquired schema lock
0924 19:57:40.162302 (+ 23us) tablet.cc:662] Decoding operations
0924 19:57:40.163354 (+ 1052us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:40.163455 (+ 101us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:40.163486 (+ 31us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:40.163825 (+ 339us) tablet.cc:701] Row locks acquired
0924 19:57:40.163885 (+ 60us) write_op.cc:260] PREPARE: finished
0924 19:57:40.164146 (+ 261us) write_op.cc:270] Start()
0924 19:57:40.164348 (+ 202us) write_op.cc:276] Timestamp: P: 1790279860164036 usec, L: 0
0924 19:57:40.164372 (+ 24us) op_driver.cc:348] REPLICATION: starting
0924 19:57:40.165175 (+ 803us) log.cc:844] Serialized 405 byte log entry
0924 19:57:40.550752 (+385577us) op_driver.cc:464] REPLICATION: finished
0924 19:57:40.567883 (+ 17131us) write_op.cc:301] APPLY: starting
0924 19:57:40.567945 (+ 62us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:40.568045 (+ 100us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:40.568164 (+ 119us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:40.568340 (+ 176us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:40.568475 (+ 135us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:40.568521 (+ 46us) write_op.cc:312] APPLY: finished
0924 19:57:40.568708 (+ 187us) log.cc:844] Serialized 23 byte log entry
0924 19:57:40.568867 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:40.568921 (+ 54us) write_op.cc:454] Released schema lock
0924 19:57:40.568950 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":27014,"child_traces":[["op",{"apply.queue_time_us":16964,"apply.run_cpu_time_us":1284,"apply.run_wall_time_us":1284,"num_ops":1,"prepare.queue_time_us":458,"prepare.run_cpu_time_us":2984,"prepare.run_wall_time_us":5012,"raft.queue_time_us":92,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":12,"replication_time_us":386267,"thread_start_us":18291,"threads_started":1}]]}
I20260924 19:57:42.408099 18910 rpcz_store.cc:276] 0924 19:57:39.945418 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.983855 (+ 38437us) service_pool.cc:224] Handling call
0924 19:57:42.396146 (+2412291us) ts_tablet_manager.cc:1151] Deleting tablet 349c6e9413bb41268d0c41d549203e2a
0924 19:57:42.396195 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:42.396375 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2411511}
I20260924 19:57:42.416437 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 175) took 2471 ms. Trace:
I20260924 19:57:42.416808 18910 rpcz_store.cc:276] 0924 19:57:39.945096 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.987342 (+ 42246us) service_pool.cc:224] Handling call
0924 19:57:42.416144 (+2428802us) ts_tablet_manager.cc:1151] Deleting tablet a2e4519aed7642678db3a518a89778de
0924 19:57:42.416189 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:42.416374 (+ 185us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2427796}
I20260924 19:57:42.416865 17612 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_6115a46b8e7444e6899712d31f41390a" } modify_external_catalogs: true
I20260924 19:57:42.425989 18224 tablet_bootstrap.cc:654] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:42.428283 18223 raft_consensus.cc:359] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.429028 18223 raft_consensus.cc:385] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:42.429322 18223 raft_consensus.cc:740] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:42.429966 18223 consensus_queue.cc:260] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.365213 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:42.431747 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_a7d63063e7ee4493abe8c76da6d4f99d [id=be19c62729a34ae1a15966662303a869]). Expected version 2 got 1
I20260924 19:57:42.432564 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:42.433666 17610 catalog_manager.cc:5838] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:42.441362 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:42.456669 19051 ts_tablet_manager.cc:1916] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:42.443485 17612 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_6115a46b8e7444e6899712d31f41390a" } modify_external_catalogs: true
I20260924 19:57:42.417961 18910 tablet_replica.cc:333] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:42.476334 18910 raft_consensus.cc:2243] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:42.479574 18910 raft_consensus.cc:2272] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:42.479816 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:42.481110 18223 ts_tablet_manager.cc:1434] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.351s user 0.007s sys 0.001s
I20260924 19:57:42.488795 18223 tablet_bootstrap.cc:492] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:42.466295 17606 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 30) took 1758 ms (client timeout 1000 ms). Trace:
I20260924 19:57:42.491814 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
I20260924 19:57:42.453503 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]) successfully deleted
W20260924 19:57:42.493711 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 349c6e9413bb41268d0c41d549203e2a because the tablet was not found. No further retry: Not found: Tablet not found: 349c6e9413bb41268d0c41d549203e2a
W20260924 19:57:42.496372 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet a2e4519aed7642678db3a518a89778de because the tablet was not found. No further retry: Not found: Tablet not found: a2e4519aed7642678db3a518a89778de
W20260924 19:57:42.491902 17606 rpcz_store.cc:269] 0924 19:57:40.707745 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.757414 (+ 49669us) service_pool.cc:224] Handling call
0924 19:57:42.466094 (+1708680us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:42.196348 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:42.196787 (+ 439us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:42.196815 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:57:42.196831 (+ 16us) tablet.cc:662] Decoding operations
0924 19:57:42.197304 (+ 473us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:42.197346 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:42.197371 (+ 25us) tablet.cc:685] Acquiring locks for 3 operations
0924 19:57:42.197681 (+ 310us) tablet.cc:701] Row locks acquired
0924 19:57:42.197693 (+ 12us) write_op.cc:260] PREPARE: finished
0924 19:57:42.197820 (+ 127us) write_op.cc:270] Start()
0924 19:57:42.197911 (+ 91us) write_op.cc:276] Timestamp: P: 1790279862197734 usec, L: 0
0924 19:57:42.197925 (+ 14us) op_driver.cc:348] REPLICATION: starting
0924 19:57:42.198500 (+ 575us) log.cc:844] Serialized 1284 byte log entry
0924 19:57:42.218838 (+ 20338us) op_driver.cc:464] REPLICATION: finished
0924 19:57:42.219218 (+ 380us) write_op.cc:301] APPLY: starting
0924 19:57:42.219282 (+ 64us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:42.219434 (+ 152us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:42.219553 (+ 119us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:42.219888 (+ 335us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:42.220047 (+ 159us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0924 19:57:42.220095 (+ 48us) write_op.cc:312] APPLY: finished
0924 19:57:42.220300 (+ 205us) log.cc:844] Serialized 35 byte log entry
0924 19:57:42.220447 (+ 147us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.220538 (+ 91us) write_op.cc:454] Released schema lock
0924 19:57:42.220578 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:42.220858 (+ 280us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.220882 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":39,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":191,"apply.run_cpu_time_us":1602,"apply.run_wall_time_us":1601,"num_ops":3,"prepare.queue_time_us":101,"prepare.run_cpu_time_us":2457,"prepare.run_wall_time_us":2457,"raft.queue_time_us":1316,"raft.run_cpu_time_us":610,"raft.run_wall_time_us":610,"replication_time_us":20801}]]}
W20260924 19:57:42.502502 17599 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6bd442f16492495f84bb8363b9bade97 consensus state has no leader
I20260924 19:57:42.505581 18231 ts_tablet_manager.cc:1929] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:42.505978 18231 log.cc:1199] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/8a524bad1c4040e58ad84fcf5a4fc47d
I20260924 19:57:42.506968 18231 ts_tablet_manager.cc:1950] T 8a524bad1c4040e58ad84fcf5a4fc47d P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:42.511533 17607 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 218) took 1750 ms. Trace:
I20260924 19:57:42.512387 18220 tablet_bootstrap.cc:654] T 6bd442f16492495f84bb8363b9bade97 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:42.511826 17607 rpcz_store.cc:276] 0924 19:57:40.760625 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.237651 (+1477026us) service_pool.cc:224] Handling call
0924 19:57:42.389493 (+151842us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:42.392647 (+ 3154us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:42.510684 (+118037us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:42.511002 (+ 318us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:42.511332 (+ 330us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:42.399798 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:42.400323 (+ 525us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:42.400389 (+ 66us) write_op.cc:435] Acquired schema lock
0924 19:57:42.400419 (+ 30us) tablet.cc:662] Decoding operations
0924 19:57:42.400812 (+ 393us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:42.400881 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:42.400911 (+ 30us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:42.401258 (+ 347us) tablet.cc:701] Row locks acquired
0924 19:57:42.401276 (+ 18us) write_op.cc:260] PREPARE: finished
0924 19:57:42.401425 (+ 149us) write_op.cc:270] Start()
0924 19:57:42.401546 (+ 121us) write_op.cc:276] Timestamp: P: 1790279862401298 usec, L: 0
0924 19:57:42.401577 (+ 31us) op_driver.cc:348] REPLICATION: starting
0924 19:57:42.402523 (+ 946us) log.cc:844] Serialized 433 byte log entry
0924 19:57:42.507636 (+105113us) op_driver.cc:464] REPLICATION: finished
0924 19:57:42.508526 (+ 890us) write_op.cc:301] APPLY: starting
0924 19:57:42.508600 (+ 74us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:42.508741 (+ 141us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:42.508869 (+ 128us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:42.509377 (+ 508us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:42.509578 (+ 201us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:42.509630 (+ 52us) write_op.cc:312] APPLY: finished
0924 19:57:42.509841 (+ 211us) log.cc:844] Serialized 29 byte log entry
0924 19:57:42.509999 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.510064 (+ 65us) write_op.cc:454] Released schema lock
0924 19:57:42.510105 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:42.510350 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.510373 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":37632,"child_traces":[["op",{"apply.queue_time_us":671,"apply.run_cpu_time_us":1814,"apply.run_wall_time_us":1814,"num_ops":2,"prepare.queue_time_us":5175,"prepare.run_cpu_time_us":3243,"prepare.run_wall_time_us":6114,"raft.queue_time_us":149,"raft.run_cpu_time_us":863,"raft.run_wall_time_us":863,"replication_time_us":105927}]]}
I20260924 19:57:42.517912 17607 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_1b2f1b739e234953861c274a098c2e1c" } modify_external_catalogs: true
I20260924 19:57:42.518432 19051 ts_tablet_manager.cc:1929] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:42.518549 17607 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_1b2f1b739e234953861c274a098c2e1c" } modify_external_catalogs: true
I20260924 19:57:42.526867 19067 tablet.cc:1751] T dd434e38c9364f55bd18383149015028 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:42.535441 19051 log.cc:1199] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/f90a677e94034c73ab5452554862c1c3
I20260924 19:57:42.536456 19051 ts_tablet_manager.cc:1950] T f90a677e94034c73ab5452554862c1c3 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:42.539311 19051 tablet_replica.cc:333] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:42.547921 19051 raft_consensus.cc:2243] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:42.548573 19051 raft_consensus.cc:2272] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:42.550640 19051 ts_tablet_manager.cc:1916] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:42.556118 18224 tablet_bootstrap.cc:492] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:42.563066 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 177) took 2693 ms. Trace:
I20260924 19:57:42.564849 17609 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 220) took 1735 ms. Trace:
I20260924 19:57:42.566382 18231 rpcz_store.cc:276] 0924 19:57:39.869705 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.869833 (+ 128us) service_pool.cc:224] Handling call
0924 19:57:42.167110 (+2297277us) ts_tablet_manager.cc:1151] Deleting tablet 8a524bad1c4040e58ad84fcf5a4fc47d
0924 19:57:42.167169 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:42.303065 (+135896us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:42.505438 (+202373us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:42.562992 (+ 57554us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2296778}
I20260924 19:57:42.569130 17613 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 216) took 1938 ms. Trace:
I20260924 19:57:42.569545 17613 rpcz_store.cc:276] 0924 19:57:40.630257 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.137939 (+1507682us) service_pool.cc:224] Handling call
0924 19:57:42.389825 (+251886us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:42.394983 (+ 5158us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:42.567989 (+173006us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:42.568478 (+ 489us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:42.568923 (+ 445us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:42.409524 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:42.417396 (+ 7872us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:42.417427 (+ 31us) write_op.cc:435] Acquired schema lock
0924 19:57:42.417444 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:42.417771 (+ 327us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:42.417812 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:42.417836 (+ 24us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:42.418075 (+ 239us) tablet.cc:701] Row locks acquired
0924 19:57:42.418089 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:42.418220 (+ 131us) write_op.cc:270] Start()
0924 19:57:42.418333 (+ 113us) write_op.cc:276] Timestamp: P: 1790279862418113 usec, L: 0
0924 19:57:42.418353 (+ 20us) op_driver.cc:348] REPLICATION: starting
0924 19:57:42.418977 (+ 624us) log.cc:844] Serialized 433 byte log entry
0924 19:57:42.561903 (+142926us) op_driver.cc:464] REPLICATION: finished
0924 19:57:42.564314 (+ 2411us) write_op.cc:301] APPLY: starting
0924 19:57:42.564372 (+ 58us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:42.564493 (+ 121us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:42.564594 (+ 101us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:42.565000 (+ 406us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:42.565127 (+ 127us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:42.565166 (+ 39us) write_op.cc:312] APPLY: finished
0924 19:57:42.565336 (+ 170us) log.cc:844] Serialized 29 byte log entry
0924 19:57:42.565485 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.565542 (+ 57us) write_op.cc:454] Released schema lock
0924 19:57:42.565575 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2292,"apply.run_cpu_time_us":1444,"apply.run_wall_time_us":1445,"mutex_wait_us":9662,"num_ops":2,"prepare.queue_time_us":12971,"prepare.run_cpu_time_us":2349,"prepare.run_wall_time_us":10620,"replication_time_us":143480,"threads_started":1}]]}
I20260924 19:57:42.565172 17609 rpcz_store.cc:276] 0924 19:57:40.829625 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.277706 (+1448081us) service_pool.cc:224] Handling call
0924 19:57:42.389241 (+111535us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:42.393054 (+ 3813us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:42.564019 (+170965us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:42.564313 (+ 294us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:42.564680 (+ 367us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:42.406001 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:42.406448 (+ 447us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:42.406474 (+ 26us) write_op.cc:435] Acquired schema lock
0924 19:57:42.406491 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:42.406867 (+ 376us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:42.406906 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:42.406937 (+ 31us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:42.407297 (+ 360us) tablet.cc:701] Row locks acquired
0924 19:57:42.407322 (+ 25us) write_op.cc:260] PREPARE: finished
0924 19:57:42.407462 (+ 140us) write_op.cc:270] Start()
0924 19:57:42.407626 (+ 164us) write_op.cc:276] Timestamp: P: 1790279862407354 usec, L: 0
0924 19:57:42.407642 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:42.408329 (+ 687us) log.cc:844] Serialized 433 byte log entry
0924 19:57:42.561659 (+153330us) op_driver.cc:464] REPLICATION: finished
0924 19:57:42.562484 (+ 825us) write_op.cc:301] APPLY: starting
0924 19:57:42.562542 (+ 58us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:42.562649 (+ 107us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:42.562755 (+ 106us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:42.563149 (+ 394us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:42.563306 (+ 157us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:42.563350 (+ 44us) write_op.cc:312] APPLY: finished
0924 19:57:42.563544 (+ 194us) log.cc:844] Serialized 29 byte log entry
0924 19:57:42.563705 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.563768 (+ 63us) write_op.cc:454] Released schema lock
0924 19:57:42.563808 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:42.564062 (+ 254us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.564089 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":1536128,"child_traces":[["op",{"apply.queue_time_us":670,"apply.run_cpu_time_us":1554,"apply.run_wall_time_us":1555,"mutex_wait_us":5021,"num_ops":2,"prepare.queue_time_us":11466,"prepare.run_cpu_time_us":2273,"prepare.run_wall_time_us":3383,"replication_time_us":153898}]]}
I20260924 19:57:42.582072 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 8a524bad1c4040e58ad84fcf5a4fc47d (table default.Table_12a91a9b08e840a3bf63db9781fcf701 [id=a3383cc128114825b7848ee8e2092211]) successfully deleted
I20260924 19:57:42.588183 17611 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 219) took 1766 ms. Trace:
I20260924 19:57:42.588526 17611 rpcz_store.cc:276] 0924 19:57:40.821833 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.247257 (+1425424us) service_pool.cc:224] Handling call
0924 19:57:42.390210 (+142953us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:42.395672 (+ 5462us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:42.587393 (+191721us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:42.587655 (+ 262us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:42.588021 (+ 366us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:42.420327 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:42.420795 (+ 468us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:42.420824 (+ 29us) write_op.cc:435] Acquired schema lock
0924 19:57:42.420845 (+ 21us) tablet.cc:662] Decoding operations
0924 19:57:42.421163 (+ 318us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:42.421201 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:42.421231 (+ 30us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:42.421469 (+ 238us) tablet.cc:701] Row locks acquired
0924 19:57:42.421489 (+ 20us) write_op.cc:260] PREPARE: finished
0924 19:57:42.421607 (+ 118us) write_op.cc:270] Start()
0924 19:57:42.421700 (+ 93us) write_op.cc:276] Timestamp: P: 1790279862421504 usec, L: 0
0924 19:57:42.421717 (+ 17us) op_driver.cc:348] REPLICATION: starting
0924 19:57:42.422239 (+ 522us) log.cc:844] Serialized 433 byte log entry
0924 19:57:42.583484 (+161245us) op_driver.cc:464] REPLICATION: finished
0924 19:57:42.584449 (+ 965us) write_op.cc:301] APPLY: starting
0924 19:57:42.584509 (+ 60us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:42.584611 (+ 102us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:42.584702 (+ 91us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:42.585101 (+ 399us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:42.586182 (+ 1081us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:42.586227 (+ 45us) write_op.cc:312] APPLY: finished
0924 19:57:42.586435 (+ 208us) log.cc:844] Serialized 29 byte log entry
0924 19:57:42.587080 (+ 645us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.587140 (+ 60us) write_op.cc:454] Released schema lock
0924 19:57:42.587201 (+ 61us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":770,"mutex_wait_us":19501,"num_ops":2,"prepare.queue_time_us":23406,"prepare.run_cpu_time_us":2139,"prepare.run_wall_time_us":3006,"replication_time_us":161599}]]}
W20260924 19:57:42.595774 17614 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f43f2a255c644ca6a68a8d867798fc1a consensus state has no leader
I20260924 19:57:42.596575 17614 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 221) took 1636 ms. Trace:
I20260924 19:57:42.597625 17615 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet ae7a0660795e4e4b87d0959ca84d5d1b
I20260924 19:57:42.599073 17615 catalog_manager.cc:6934] ReplaceTablet: tablet ae7a0660795e4e4b87d0959ca84d5d1b deleted and replaced by tablet bea1ebdf647e4ef9a131a0398672b8ab
I20260924 19:57:42.599493 17615 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51280 (request call id 222) took 1397 ms. Trace:
I20260924 19:57:42.597225 17614 rpcz_store.cc:276] 0924 19:57:40.960585 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.372459 (+1411874us) service_pool.cc:224] Handling call
0924 19:57:42.397735 (+ 25276us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:42.398651 (+ 916us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:42.438951 (+ 40300us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:42.439238 (+ 287us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:42.595003 (+155765us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:42.596510 (+ 1507us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:42.452672 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:42.453143 (+ 471us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:42.453173 (+ 30us) write_op.cc:435] Acquired schema lock
0924 19:57:42.453188 (+ 15us) tablet.cc:662] Decoding operations
0924 19:57:42.453531 (+ 343us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:42.453574 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:42.453598 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:42.453779 (+ 181us) tablet.cc:701] Row locks acquired
0924 19:57:42.453800 (+ 21us) write_op.cc:260] PREPARE: finished
0924 19:57:42.453942 (+ 142us) write_op.cc:270] Start()
0924 19:57:42.454037 (+ 95us) write_op.cc:276] Timestamp: P: 1790279862453829 usec, L: 0
0924 19:57:42.454055 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:42.454617 (+ 562us) log.cc:844] Serialized 405 byte log entry
0924 19:57:42.583817 (+129200us) op_driver.cc:464] REPLICATION: finished
0924 19:57:42.592924 (+ 9107us) write_op.cc:301] APPLY: starting
0924 19:57:42.592979 (+ 55us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:42.593070 (+ 91us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:42.593168 (+ 98us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:42.593327 (+ 159us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:42.593455 (+ 128us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:42.593494 (+ 39us) write_op.cc:312] APPLY: finished
0924 19:57:42.593701 (+ 207us) log.cc:844] Serialized 23 byte log entry
0924 19:57:42.594651 (+ 950us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.594717 (+ 66us) write_op.cc:454] Released schema lock
0924 19:57:42.594766 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:42.595003 (+ 237us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.595027 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":8979,"apply.run_cpu_time_us":1260,"apply.run_wall_time_us":2045,"num_ops":1,"prepare.queue_time_us":12106,"prepare.run_cpu_time_us":2192,"prepare.run_wall_time_us":2192,"replication_time_us":129686}]]}
I20260924 19:57:42.600181 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet ae7a0660795e4e4b87d0959ca84d5d1b with delete_type TABLET_DATA_DELETED (replaced by tablet bea1ebdf647e4ef9a131a0398672b8ab) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:42.601013 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 178) took 2728 ms. Trace:
I20260924 19:57:42.601419 18231 rpcz_store.cc:276] 0924 19:57:39.872376 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:39.872511 (+ 135us) service_pool.cc:224] Handling call
0924 19:57:42.600686 (+2728175us) ts_tablet_manager.cc:1151] Deleting tablet d970cc92a1134a98a613d198a319cb01
0924 19:57:42.600731 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:42.600944 (+ 213us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2726915}
I20260924 19:57:42.599747 17615 rpcz_store.cc:276] 0924 19:57:41.202131 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.405341 (+1203210us) service_pool.cc:224] Handling call
0924 19:57:42.599441 (+194100us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:42.454968 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:42.455438 (+ 470us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:42.455463 (+ 25us) write_op.cc:435] Acquired schema lock
0924 19:57:42.455479 (+ 16us) tablet.cc:662] Decoding operations
0924 19:57:42.455805 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:42.455851 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:42.455889 (+ 38us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:42.456110 (+ 221us) tablet.cc:701] Row locks acquired
0924 19:57:42.456127 (+ 17us) write_op.cc:260] PREPARE: finished
0924 19:57:42.456243 (+ 116us) write_op.cc:270] Start()
0924 19:57:42.456325 (+ 82us) write_op.cc:276] Timestamp: P: 1790279862456136 usec, L: 0
0924 19:57:42.456343 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:42.456849 (+ 506us) log.cc:844] Serialized 625 byte log entry
0924 19:57:42.584016 (+127167us) op_driver.cc:464] REPLICATION: finished
0924 19:57:42.595268 (+ 11252us) write_op.cc:301] APPLY: starting
0924 19:57:42.595316 (+ 48us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:42.595437 (+ 121us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:42.595533 (+ 96us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:42.595941 (+ 408us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:42.596149 (+ 208us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:42.596191 (+ 42us) write_op.cc:312] APPLY: finished
0924 19:57:42.596367 (+ 176us) log.cc:844] Serialized 29 byte log entry
0924 19:57:42.596926 (+ 559us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.596989 (+ 63us) write_op.cc:454] Released schema lock
0924 19:57:42.597023 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":11137,"num_ops":2,"prepare.queue_time_us":11587,"prepare.run_cpu_time_us":2097,"prepare.run_wall_time_us":2096,"replication_time_us":127603,"threads_started":1}]]}
W20260924 19:57:42.602967 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet d970cc92a1134a98a613d198a319cb01 because the tablet was not found. No further retry: Not found: Tablet not found: d970cc92a1134a98a613d198a319cb01
I20260924 19:57:42.603920 18910 ts_tablet_manager.cc:1916] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:42.609900 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet ae7a0660795e4e4b87d0959ca84d5d1b with delete_type TABLET_DATA_DELETED (replaced by tablet bea1ebdf647e4ef9a131a0398672b8ab) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:42.610538 19073 tablet.cc:1751] T dd434e38c9364f55bd18383149015028 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:42.605404 18224 ts_tablet_manager.cc:1403] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.257s user 0.010s sys 0.005s
I20260924 19:57:42.628943 18224 raft_consensus.cc:359] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.629666 18224 raft_consensus.cc:385] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:42.629946 18224 raft_consensus.cc:740] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:42.636142 17615 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_3b871d019f0d40c4b916ef34227f5730" } new_table_name: "default.Table_88352246addd43f4be871c983413b2de" modify_external_catalogs: true
I20260924 19:57:42.645741 18780 raft_consensus.cc:493] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:42.646858 18780 raft_consensus.cc:515] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.649235 18220 tablet_bootstrap.cc:492] T 6bd442f16492495f84bb8363b9bade97 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:42.652073 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df722b0bbfd04ce2943f0e5b8058c50f" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:42.653630 17753 leader_election.cc:343] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:42.654309 18780 leader_election.cc:290] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:42.656404 18220 ts_tablet_manager.cc:1403] T 6bd442f16492495f84bb8363b9bade97 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.271s user 0.019s sys 0.004s
W20260924 19:57:42.662554 19025 log.cc:927] Time spent T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Append to log took a long time: real 0.085s user 0.000s sys 0.002s
I20260924 19:57:42.671428 18231 tablet_replica.cc:333] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:42.672382 18231 raft_consensus.cc:2243] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:42.673027 18231 raft_consensus.cc:2272] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:42.675071 18231 ts_tablet_manager.cc:1916] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:42.707893 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:42.708843 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]). Expected version 1 got 0
I20260924 19:57:42.700553 18220 raft_consensus.cc:359] T 6bd442f16492495f84bb8363b9bade97 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.710016 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_a7d63063e7ee4493abe8c76da6d4f99d [id=be19c62729a34ae1a15966662303a869]). Expected version 2 got 1
I20260924 19:57:42.710721 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:42.711300 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:42.710525 18220 raft_consensus.cc:385] T 6bd442f16492495f84bb8363b9bade97 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:42.715489 18224 consensus_queue.cc:260] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.725232 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df3dcf4ddcf4efa9aa047da56dfb7a7" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:42.733160 17816 raft_consensus.cc:2468] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:42.744545 18224 ts_tablet_manager.cc:1434] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.118s user 0.005s sys 0.000s
I20260924 19:57:42.714805 18220 raft_consensus.cc:740] T 6bd442f16492495f84bb8363b9bade97 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:42.754344 18220 consensus_queue.cc:260] T 6bd442f16492495f84bb8363b9bade97 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.719962 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_b2824cbbb81c4186840b758c61eccbb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:42.788237 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2824cbbb81c4186840b758c61eccbb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:42.759042 17611 master_service.cc:960] ReplaceTablet: received request to replace tablet 44199d2655f14dcf97afe2180789aae4 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:42.778466 17609 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_a57608cee13f4af2b03aa54baa4acb7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:42.796437 17609 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a57608cee13f4af2b03aa54baa4acb7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:42.779476 17613 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_65d20bb187ea4069b2e589e7643028be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:42.801292 17613 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65d20bb187ea4069b2e589e7643028be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:42.722047 18051 leader_election.cc:304] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:42.810660 19047 raft_consensus.cc:2804] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:42.811189 19047 raft_consensus.cc:697] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:42.765743 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df722b0bbfd04ce2943f0e5b8058c50f" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:42.812247 19047 consensus_queue.cc:237] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:42.788494 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 59231.9us
I20260924 19:57:42.799605 18951 raft_consensus.cc:493] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:42.814307 18951 raft_consensus.cc:515] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.801519 18223 tablet_bootstrap.cc:654] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:42.815807 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 94126.2us
I20260924 19:57:42.819402 18220 ts_tablet_manager.cc:1434] T 6bd442f16492495f84bb8363b9bade97 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.161s user 0.010s sys 0.001s
I20260924 19:57:42.820363 18220 tablet_bootstrap.cc:492] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:42.822780 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_83714385573a44b2af65ac658fbe2cfa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:42.824661 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83714385573a44b2af65ac658fbe2cfa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:42.842696 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
I20260924 19:57:42.844785 18224 tablet_bootstrap.cc:492] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:42.835337 18951 leader_election.cc:290] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:42.846289 18051 leader_election.cc:304] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:42.847098 18951 raft_consensus.cc:2804] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:42.847505 18951 raft_consensus.cc:493] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:42.847862 18951 raft_consensus.cc:3060] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:42.851182 18050 consensus_peers.cc:597] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:42.852466 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e6d926ff7c24901a15b0c7b2239fbad" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:42.853216 17816 raft_consensus.cc:2468] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:42.854655 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 3 got 0
I20260924 19:57:42.859083 17608 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]). Expected version 1 got 0
W20260924 19:57:42.856144 19025 log.cc:927] Time spent T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Append to log took a long time: real 0.149s user 0.003s sys 0.001s
I20260924 19:57:42.867848 17967 raft_consensus.cc:3060] T 3b328d37d55e4116a29322caaa21d431 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:42.884346 17522 leader_election.cc:336] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309): Timed out: RequestConsensusVote RPC to 127.17.2.190:46309 timed out after 0.627s (SENT)
I20260924 19:57:42.884793 17522 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:42.886955 18051 leader_election.cc:304] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:42.889964 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e6d926ff7c24901a15b0c7b2239fbad" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:42.891813 19048 raft_consensus.cc:2804] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:42.892168 19048 raft_consensus.cc:493] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:42.892551 19048 raft_consensus.cc:3060] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:42.893200 17970 raft_consensus.cc:3060] T 6bd442f16492495f84bb8363b9bade97 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:42.898723 17753 leader_election.cc:343] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:42.899257 17753 leader_election.cc:304] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:42.899521 18050 leader_election.cc:343] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:42.899968 19069 raft_consensus.cc:2749] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:42.905172 18951 raft_consensus.cc:515] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.914109 18090 tablet_service.cc:1558] Processing DeleteTablet for tablet ae7a0660795e4e4b87d0959ca84d5d1b with delete_type TABLET_DATA_DELETED (replaced by tablet bea1ebdf647e4ef9a131a0398672b8ab) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:42.865711 17613 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_fbe470367ca848268ffa88adbee30894" } new_table_name: "default.Table_a12f8684de2041ec83f2fee77829afdd" modify_external_catalogs: true
I20260924 19:57:42.912405 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet f90a677e94034c73ab5452554862c1c3 (table default.Table_fca275d61c914e72b792ca0bb116d0bf [id=19596b2bff2e46399e96dcc6d5fbfbed]) successfully deleted
W20260924 19:57:42.946326 17522 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.190:46309, user_credentials={real_user=slave}} blocked reactor thread for 61983.6us
I20260924 19:57:42.951656 18220 tablet_bootstrap.cc:654] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:42.952224 17612 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:57:42.954597 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet ba46ac86b242407abd3e8dbf09f12efa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:42.955832 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet ba46ac86b242407abd3e8dbf09f12efa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:42.963738 18780 raft_consensus.cc:493] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:42.964349 18780 raft_consensus.cc:515] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.966059 19048 raft_consensus.cc:515] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:42.967793 18780 leader_election.cc:290] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:42.967939 17607 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 44199d2655f14dcf97afe2180789aae4
I20260924 19:57:42.970738 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 44199d2655f14dcf97afe2180789aae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:42.985437 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aef0359a77240679240823612507ee8" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:42.986888 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet 44199d2655f14dcf97afe2180789aae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:42.989686 17753 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:42.990381 19086 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 19:57:43.004575 18910 ts_tablet_manager.cc:1929] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:43.009359 18965 consensus_queue.cc:1048] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:43.025174 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_5e789131162643caa4c014a55244066f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:43.025873 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aef0359a77240679240823612507ee8" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:43.026741 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e789131162643caa4c014a55244066f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:43.029989 17753 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:43.030439 17753 leader_election.cc:304] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:43.031255 18780 raft_consensus.cc:2749] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:43.037989 17613 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d129e0979a584f40ba9a975349cf15a5 consensus state has no leader
I20260924 19:57:43.039768 18224 tablet_bootstrap.cc:700] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202: Time spent opening tablet: real 0.165s user 0.003s sys 0.000s
I20260924 19:57:43.040959 18224 tablet_bootstrap.cc:654] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:43.063859 17905 consensus_peers.cc:597] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:43.070994 17613 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_8be4bd6989c649469efa4d12976b4576"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:43.072316 17905 consensus_peers.cc:597] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f90a677e94034c73ab5452554862c1c3. This is attempt 1: this message will repeat every 5th retry.
W20260924 19:57:43.072472 17613 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8be4bd6989c649469efa4d12976b4576 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:43.068723 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_1d9dfc2ce03046ba955d3d82049402fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:43.076670 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d9dfc2ce03046ba955d3d82049402fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:43.087335 18910 log.cc:1199] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/dd2f4eb8bf254c4e85b6a9072271766b
I20260924 19:57:43.094587 18220 tablet_bootstrap.cc:492] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:43.107355 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e6d926ff7c24901a15b0c7b2239fbad" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:43.109527 18910 ts_tablet_manager.cc:1950] T dd2f4eb8bf254c4e85b6a9072271766b P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:43.112036 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_0673b53c69bf4c429697ba739ef113e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:43.113546 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0673b53c69bf4c429697ba739ef113e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:43.113662 18220 ts_tablet_manager.cc:1403] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.293s user 0.008s sys 0.011s
I20260924 19:57:43.121938 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 179) took 2934 ms. Trace:
I20260924 19:57:43.124972 18224 tablet_bootstrap.cc:492] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:43.125478 18224 ts_tablet_manager.cc:1403] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.287s user 0.009s sys 0.001s
I20260924 19:57:43.126011 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet ba46ac86b242407abd3e8dbf09f12efa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:43.128455 18224 raft_consensus.cc:359] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.129113 18224 raft_consensus.cc:385] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:43.129364 18224 raft_consensus.cc:740] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:43.129967 18224 consensus_queue.cc:260] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.131934 18224 ts_tablet_manager.cc:1434] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 19:57:43.132793 18224 tablet_bootstrap.cc:492] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:43.137550 18223 tablet_bootstrap.cc:492] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:43.125521 19083 tablet.cc:1751] T dd434e38c9364f55bd18383149015028 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:43.138271 18951 leader_election.cc:290] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:43.144207 19048 leader_election.cc:290] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:43.148639 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df3dcf4ddcf4efa9aa047da56dfb7a7" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:43.149178 17821 raft_consensus.cc:3060] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:43.152359 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df3dcf4ddcf4efa9aa047da56dfb7a7" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:43.156894 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e6d926ff7c24901a15b0c7b2239fbad" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:43.157536 17817 raft_consensus.cc:3060] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:43.159147 18224 tablet_bootstrap.cc:654] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:43.165958 17817 raft_consensus.cc:2468] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:43.169338 18051 leader_election.cc:304] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:43.166771 17821 raft_consensus.cc:2468] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:43.173985 19080 raft_consensus.cc:2804] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:43.174485 19080 raft_consensus.cc:697] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:43.175384 19080 consensus_queue.cc:237] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.178797 18051 leader_election.cc:304] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:43.180665 17614 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_9716bbadd5a3442382dd3562939de32d" } modify_external_catalogs: true
I20260924 19:57:43.181202 17614 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_9716bbadd5a3442382dd3562939de32d" } modify_external_catalogs: true
I20260924 19:57:43.183045 17609 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_d944ee03376e4f54ae842e8c7dfb5b5c" } modify_external_catalogs: true
I20260924 19:57:43.183643 18224 tablet_bootstrap.cc:492] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:43.183554 17609 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_d944ee03376e4f54ae842e8c7dfb5b5c" } modify_external_catalogs: true
I20260924 19:57:43.184058 18224 ts_tablet_manager.cc:1403] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.004s
I20260924 19:57:43.178673 18220 raft_consensus.cc:359] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:43.184926 18220 raft_consensus.cc:385] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:43.185241 18220 raft_consensus.cc:740] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:43.186585 18224 raft_consensus.cc:359] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.187227 18224 raft_consensus.cc:385] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:43.187495 18224 raft_consensus.cc:740] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:43.188131 18224 consensus_queue.cc:260] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.190176 18224 ts_tablet_manager.cc:1434] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 19:57:43.194164 18965 raft_consensus.cc:2804] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:43.194269 18082 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38450 (request call id 187) took 1276 ms. Trace:
I20260924 19:57:43.194535 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet 44199d2655f14dcf97afe2180789aae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:43.195071 18223 ts_tablet_manager.cc:1403] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.706s user 0.003s sys 0.017s
I20260924 19:57:43.194736 18082 rpcz_store.cc:276] 0924 19:57:41.901666 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:41.907962 (+ 6296us) service_pool.cc:224] Handling call
0924 19:57:41.909311 (+ 1349us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0924 19:57:41.909388 (+ 77us) ts_tablet_manager.cc:682] Creating new metadata...
0924 19:57:42.759772 (+850384us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.037692 (+277920us) tablet_replica.cc:206] Creating consensus instance
0924 19:57:43.178300 (+140608us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:43.202695 19051 ts_tablet_manager.cc:1929] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:43.191053 18224 tablet_bootstrap.cc:492] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:43.212878 18224 tablet_bootstrap.cc:654] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:43.205251 18050 leader_election.cc:343] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:43.205708 18965 raft_consensus.cc:697] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:43.215596 19051 log.cc:1199] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/910d61b8c3574302b5a328187226571e
I20260924 19:57:43.216851 18965 consensus_queue.cc:237] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.218093 19051 ts_tablet_manager.cc:1950] T 910d61b8c3574302b5a328187226571e P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
W20260924 19:57:43.218559 19025 log.cc:927] Time spent T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20260924 19:57:43.185978 18220 consensus_queue.cc:260] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:43.219509 19025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=1 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260924 19:57:43.230685 18220 ts_tablet_manager.cc:1434] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.111s user 0.003s sys 0.006s
I20260924 19:57:43.231699 18220 tablet_bootstrap.cc:492] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:43.221280 18910 rpcz_store.cc:276] 0924 19:57:40.189700 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.190119 (+ 419us) service_pool.cc:224] Handling call
0924 19:57:42.417791 (+2227672us) ts_tablet_manager.cc:1151] Deleting tablet dd2f4eb8bf254c4e85b6a9072271766b
0924 19:57:42.417837 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:42.735406 (+317569us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.004420 (+269014us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.121873 (+117453us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2226227}
I20260924 19:57:43.240795 18910 tablet_replica.cc:333] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:43.241788 18910 raft_consensus.cc:2243] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:43.242470 18910 raft_consensus.cc:2272] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:43.242775 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet dd2f4eb8bf254c4e85b6a9072271766b (table default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b]) successfully deleted
I20260924 19:57:43.245800 18231 ts_tablet_manager.cc:1929] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:43.246316 18231 log.cc:1199] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/dd2f4eb8bf254c4e85b6a9072271766b
I20260924 19:57:43.247186 18231 ts_tablet_manager.cc:1950] T dd2f4eb8bf254c4e85b6a9072271766b P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:43.220747 19025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Bootstrap complete.
I20260924 19:57:43.248540 18224 tablet_bootstrap.cc:492] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:43.248972 18224 ts_tablet_manager.cc:1403] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.000s
I20260924 19:57:43.211257 18223 raft_consensus.cc:359] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.249871 18223 raft_consensus.cc:385] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:43.250845 18223 raft_consensus.cc:740] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:43.251071 17613 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_97a3c9fb681344628ba099ed831d60b3" } new_table_name: "default.Table_9b8f6ad547fc4c3fa379c93590aebd56" modify_external_catalogs: true
I20260924 19:57:43.251415 18224 raft_consensus.cc:359] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.252033 18224 raft_consensus.cc:385] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:43.252305 18224 raft_consensus.cc:740] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:43.252720 18223 consensus_queue.cc:260] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.259436 18224 consensus_queue.cc:260] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.261905 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_fcbe3b81791a4aaf923847b1366eb6f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:43.264628 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcbe3b81791a4aaf923847b1366eb6f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:43.266846 18223 ts_tablet_manager.cc:1434] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.071s user 0.005s sys 0.001s
I20260924 19:57:43.267951 18224 ts_tablet_manager.cc:1434] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.019s user 0.002s sys 0.005s
W20260924 19:57:43.223134 18050 leader_election.cc:343] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:43.271572 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 48474.9us
I20260924 19:57:43.303679 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 191) took 1182 ms. Trace:
I20260924 19:57:43.313016 18220 tablet_bootstrap.cc:654] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:43.316293 19106 tablet.cc:1751] T dbb82914df65416aaf47db8e507afb40 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:43.311511 18223 tablet_bootstrap.cc:492] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:43.353550 18910 ts_tablet_manager.cc:1916] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:43.324254 19051 rpcz_store.cc:276] 0924 19:57:42.121572 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.122795 (+ 1223us) service_pool.cc:224] Handling call
0924 19:57:42.539103 (+416308us) ts_tablet_manager.cc:1151] Deleting tablet 910d61b8c3574302b5a328187226571e
0924 19:57:42.539182 (+ 79us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:42.908523 (+369341us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.202444 (+293921us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.303603 (+101159us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":415727}
W20260924 19:57:43.315698 18050 consensus_peers.cc:597] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260924 19:57:43.362522 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 46936.9us
I20260924 19:57:43.367079 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 910d61b8c3574302b5a328187226571e (table default.Table_2b157ac766d244e081168b817296bd4b [id=1bef6766d6a049a9a5126339c3f67842]) successfully deleted
I20260924 19:57:43.382156 17614 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet 4632c09468b448fb861cd845c172fe07
I20260924 19:57:43.398905 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 182) took 3260 ms. Trace:
I20260924 19:57:43.399493 18231 rpcz_store.cc:276] 0924 19:57:40.138072 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.148307 (+ 10235us) service_pool.cc:224] Handling call
0924 19:57:42.671256 (+2522949us) ts_tablet_manager.cc:1151] Deleting tablet dd2f4eb8bf254c4e85b6a9072271766b
0924 19:57:42.671307 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.005943 (+334636us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.245655 (+239712us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.398842 (+153187us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2520010}
I20260924 19:57:43.401188 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet dd2f4eb8bf254c4e85b6a9072271766b (table default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b]) successfully deleted
I20260924 19:57:43.402208 18231 tablet_replica.cc:333] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:43.403625 18224 tablet_bootstrap.cc:492] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:43.409374 18224 tablet_bootstrap.cc:654] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:43.372963 19080 consensus_queue.cc:1048] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.050s
W20260924 19:57:43.425877 17613 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d930e7f4a6b7471899dab1b28e40f93d consensus state has no leader
I20260924 19:57:43.383466 19083 tablet.cc:1751] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:43.430835 19112 tablet.cc:1751] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:43.409068 19109 tablet.cc:1751] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:43.438700 18224 tablet_bootstrap.cc:492] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:43.439131 18224 ts_tablet_manager.cc:1403] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.000s
I20260924 19:57:43.441496 18224 raft_consensus.cc:359] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.442122 18224 raft_consensus.cc:385] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:43.442360 18224 raft_consensus.cc:740] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:43.443688 18223 tablet_bootstrap.cc:700] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent opening tablet: real 0.113s user 0.004s sys 0.000s
I20260924 19:57:43.364359 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 192) took 1185 ms. Trace:
I20260924 19:57:43.455408 19106 tablet.cc:1751] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:43.459620 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_954dba173a70455b90daee9230b44b6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:43.461091 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_954dba173a70455b90daee9230b44b6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:43.461915 19051 rpcz_store.cc:276] 0924 19:57:42.179376 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.182214 (+ 2838us) service_pool.cc:224] Handling call
0924 19:57:43.363997 (+1181783us) ts_tablet_manager.cc:1151] Deleting tablet 151247d6b5254cd884c58db9311ac04a
0924 19:57:43.364077 (+ 80us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.364297 (+ 220us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134894}
I20260924 19:57:43.463797 18223 tablet_bootstrap.cc:654] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:43.466451 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_2a8be69ab9604081a07df5197e7fab9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:43.468080 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a8be69ab9604081a07df5197e7fab9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:43.472457 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 151247d6b5254cd884c58db9311ac04a because the tablet was not found. No further retry: Not found: Tablet not found: 151247d6b5254cd884c58db9311ac04a
I20260924 19:57:43.483572 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 193) took 1299 ms. Trace:
I20260924 19:57:43.439633 19111 tablet.cc:1751] T dbb82914df65416aaf47db8e507afb40 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:43.484097 19051 rpcz_store.cc:276] 0924 19:57:42.184307 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.184740 (+ 433us) service_pool.cc:224] Handling call
0924 19:57:43.483192 (+1298452us) ts_tablet_manager.cc:1151] Deleting tablet c884f12860164f12bbcd0d2f87067013
0924 19:57:43.483268 (+ 76us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.483501 (+ 233us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1241955}
I20260924 19:57:43.467458 18224 consensus_queue.cc:260] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:43.485918 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet c884f12860164f12bbcd0d2f87067013 because the tablet was not found. No further retry: Not found: Tablet not found: c884f12860164f12bbcd0d2f87067013
I20260924 19:57:43.487059 18224 ts_tablet_manager.cc:1434] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.048s user 0.003s sys 0.002s
I20260924 19:57:43.490712 18220 tablet_bootstrap.cc:492] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:43.491269 18220 ts_tablet_manager.cc:1403] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.260s user 0.013s sys 0.003s
I20260924 19:57:43.494057 17970 raft_consensus.cc:3060] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:43.508635 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 194) took 1324 ms. Trace:
I20260924 19:57:43.510715 18231 raft_consensus.cc:2243] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:43.511358 18231 raft_consensus.cc:2272] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:43.517321 19051 rpcz_store.cc:276] 0924 19:57:42.184612 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.185222 (+ 610us) service_pool.cc:224] Handling call
0924 19:57:43.508292 (+1323070us) ts_tablet_manager.cc:1151] Deleting tablet 83ee43b6dc064dc7b320691d52e57ea7
0924 19:57:43.508352 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.508570 (+ 218us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278651}
W20260924 19:57:43.518908 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 83ee43b6dc064dc7b320691d52e57ea7 because the tablet was not found. No further retry: Not found: Tablet not found: 83ee43b6dc064dc7b320691d52e57ea7
I20260924 19:57:43.520810 18231 ts_tablet_manager.cc:1916] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:43.525182 18223 tablet_bootstrap.cc:492] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:43.525619 18223 ts_tablet_manager.cc:1403] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.214s user 0.011s sys 0.000s
I20260924 19:57:43.534309 18224 tablet_bootstrap.cc:492] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:43.543812 17601 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet d930e7f4a6b7471899dab1b28e40f93d (table default.Table_9b8f6ad547fc4c3fa379c93590aebd56 [id=4fc8e565a6a640549716cee596fa3f9d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d930e7f4a6b7471899dab1b28e40f93d consensus state has no leader
I20260924 19:57:43.556352 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 196) took 1346 ms. Trace:
I20260924 19:57:43.556849 18224 tablet_bootstrap.cc:654] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:43.559213 18220 raft_consensus.cc:359] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.561324 18220 raft_consensus.cc:385] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:43.562662 18220 raft_consensus.cc:740] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:43.566694 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_6ddb9314aa114575b5317b1d08ba138e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:43.568367 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ddb9314aa114575b5317b1d08ba138e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:43.570914 17872 heartbeater.cc:344] Connected to a master server at 127.17.2.190:46309
I20260924 19:57:43.584815 18231 ts_tablet_manager.cc:1929] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260924 19:57:43.584509 19051 rpcz_store.cc:276] 0924 19:57:42.209825 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.210284 (+ 459us) service_pool.cc:224] Handling call
0924 19:57:43.556031 (+1345747us) ts_tablet_manager.cc:1151] Deleting tablet 8a524bad1c4040e58ad84fcf5a4fc47d
0924 19:57:43.556094 (+ 63us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.556297 (+ 203us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1326194}
I20260924 19:57:43.573673 18220 consensus_queue.cc:260] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.586584 18224 tablet_bootstrap.cc:492] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:43.595470 18224 ts_tablet_manager.cc:1403] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.000s
I20260924 19:57:43.597965 18224 raft_consensus.cc:359] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.598570 18224 raft_consensus.cc:385] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:43.598809 18224 raft_consensus.cc:740] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:43.599666 18224 consensus_queue.cc:260] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.601706 18224 ts_tablet_manager.cc:1434] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 19:57:43.602555 18224 tablet_bootstrap.cc:492] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:43.603293 18231 log.cc:1199] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/83ee43b6dc064dc7b320691d52e57ea7
I20260924 19:57:43.601634 18220 ts_tablet_manager.cc:1434] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.110s user 0.007s sys 0.003s
I20260924 19:57:43.606753 18231 ts_tablet_manager.cc:1950] T 83ee43b6dc064dc7b320691d52e57ea7 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:43.610445 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 183) took 2677 ms. Trace:
I20260924 19:57:43.611125 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 197) took 1401 ms. Trace:
I20260924 19:57:43.613313 19051 rpcz_store.cc:276] 0924 19:57:42.210181 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.211880 (+ 1699us) service_pool.cc:224] Handling call
0924 19:57:43.610751 (+1398871us) ts_tablet_manager.cc:1151] Deleting tablet d970cc92a1134a98a613d198a319cb01
0924 19:57:43.610822 (+ 71us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.611066 (+ 244us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380668}
W20260924 19:57:43.614786 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 8a524bad1c4040e58ad84fcf5a4fc47d because the tablet was not found. No further retry: Not found: Tablet not found: 8a524bad1c4040e58ad84fcf5a4fc47d
W20260924 19:57:43.615527 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet d970cc92a1134a98a613d198a319cb01 because the tablet was not found. No further retry: Not found: Tablet not found: d970cc92a1134a98a613d198a319cb01
I20260924 19:57:43.618938 18231 rpcz_store.cc:276] 0924 19:57:40.932868 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.933182 (+ 314us) service_pool.cc:224] Handling call
0924 19:57:43.402065 (+2468883us) ts_tablet_manager.cc:1151] Deleting tablet 83ee43b6dc064dc7b320691d52e57ea7
0924 19:57:43.402105 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.528345 (+126240us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.584681 (+ 56336us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.610379 (+ 25698us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":107585,"tablet-delete.queue_time_us":2468284}
I20260924 19:57:43.623881 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]) successfully deleted
I20260924 19:57:43.624639 18231 tablet_replica.cc:333] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:43.626617 18231 raft_consensus.cc:2243] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:43.624190 18220 tablet_bootstrap.cc:492] T affce4de8e0b412aa3b4bf19f9be7b6e P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:43.627996 18231 raft_consensus.cc:2272] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:43.628566 18223 raft_consensus.cc:359] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.629349 18223 raft_consensus.cc:385] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:43.629685 18223 raft_consensus.cc:740] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:43.630538 18223 consensus_queue.cc:260] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.634676 18231 ts_tablet_manager.cc:1916] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:43.645617 18778 raft_consensus.cc:493] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eaf64bfb64aa42948c9bc1411ce4ce35)
I20260924 19:57:43.646087 18778 raft_consensus.cc:515] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:43.644627 18224 tablet_bootstrap.cc:654] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:43.650118 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07f86f24d9d14d05898e1c0f69428850" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
W20260924 19:57:43.638247 19120 log.cc:927] Time spent T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260924 19:57:43.656636 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 198) took 1444 ms. Trace:
I20260924 19:57:43.657407 19080 raft_consensus.cc:493] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:43.662495 19080 raft_consensus.cc:515] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.665596 19080 leader_election.cc:290] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:43.659720 18965 consensus_queue.cc:1048] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:43.667886 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92c0daa7d67e4aa390e334c669adda76" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:43.668602 17821 raft_consensus.cc:2468] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:43.668870 19051 rpcz_store.cc:276] 0924 19:57:42.212627 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.212733 (+ 106us) service_pool.cc:224] Handling call
0924 19:57:43.656287 (+1443554us) ts_tablet_manager.cc:1151] Deleting tablet 2cf62a73c49245198d65011875148dc3
0924 19:57:43.656356 (+ 69us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.656576 (+ 220us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1426007}
I20260924 19:57:43.669600 18051 leader_election.cc:304] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
W20260924 19:57:43.670473 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 2cf62a73c49245198d65011875148dc3 because the tablet was not found. No further retry: Not found: Tablet not found: 2cf62a73c49245198d65011875148dc3
I20260924 19:57:43.671661 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 199) took 1458 ms. Trace:
I20260924 19:57:43.673642 18224 tablet_bootstrap.cc:492] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:43.674085 18224 ts_tablet_manager.cc:1403] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.000s
I20260924 19:57:43.673247 18965 raft_consensus.cc:2804] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:43.674484 18965 raft_consensus.cc:493] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:43.674880 18965 raft_consensus.cc:3060] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:43.676445 18224 raft_consensus.cc:359] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.677059 18224 raft_consensus.cc:385] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:43.677309 18224 raft_consensus.cc:740] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:43.683578 18778 leader_election.cc:290] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:43.684365 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92c0daa7d67e4aa390e334c669adda76" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:43.684698 18780 raft_consensus.cc:493] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:43.685151 18780 raft_consensus.cc:515] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.685669 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "07f86f24d9d14d05898e1c0f69428850" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:43.687700 17905 leader_election.cc:304] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:43.689152 19076 raft_consensus.cc:2749] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:43.687073 18780 leader_election.cc:290] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:43.690002 18780 raft_consensus.cc:493] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:43.690717 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f43f2a255c644ca6a68a8d867798fc1a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:43.692129 18050 leader_election.cc:343] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:43.692224 17753 leader_election.cc:343] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:43.692196 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f43f2a255c644ca6a68a8d867798fc1a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:43.690871 18780 raft_consensus.cc:515] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.692891 19051 rpcz_store.cc:276] 0924 19:57:42.213413 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.213538 (+ 125us) service_pool.cc:224] Handling call
0924 19:57:43.671329 (+1457791us) ts_tablet_manager.cc:1151] Deleting tablet dd2f4eb8bf254c4e85b6a9072271766b
0924 19:57:43.671381 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.671603 (+ 222us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1439824}
W20260924 19:57:43.694358 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet dd2f4eb8bf254c4e85b6a9072271766b because the tablet was not found. No further retry: Not found: Tablet not found: dd2f4eb8bf254c4e85b6a9072271766b
I20260924 19:57:43.695225 18019 heartbeater.cc:344] Connected to a master server at 127.17.2.190:46309
W20260924 19:57:43.695298 17753 leader_election.cc:343] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:43.695647 17753 leader_election.cc:304] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:43.696275 18223 ts_tablet_manager.cc:1434] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.170s user 0.003s sys 0.005s
I20260924 19:57:43.697316 18223 tablet_bootstrap.cc:492] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:43.698297 18780 leader_election.cc:290] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:43.699981 18780 raft_consensus.cc:2749] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:43.700045 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b99ff1920274831b58d8283bd592818" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:43.701292 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b99ff1920274831b58d8283bd592818" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:43.703001 17753 leader_election.cc:343] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:43.704247 17753 leader_election.cc:343] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:43.704600 17753 leader_election.cc:304] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:43.705519 18780 raft_consensus.cc:2749] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:43.711823 18224 consensus_queue.cc:260] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.720468 18224 ts_tablet_manager.cc:1434] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.046s user 0.006s sys 0.000s
I20260924 19:57:43.722003 18224 tablet_bootstrap.cc:492] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:43.723392 17607 master_service.cc:960] ReplaceTablet: received request to replace tablet ba46ac86b242407abd3e8dbf09f12efa from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:43.739590 18778 raft_consensus.cc:493] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eaf64bfb64aa42948c9bc1411ce4ce35)
I20260924 19:57:43.740110 18778 raft_consensus.cc:515] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.743995 17607 master_service.cc:960] ReplaceTablet: received request to replace tablet 3192f66a492442c1bf4137cfe7e13bc9 from {username='slave'} at 127.0.0.1:51280
W20260924 19:57:43.744292 17906 leader_election.cc:343] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: d970cc92a1134a98a613d198a319cb01
I20260924 19:57:43.744817 19125 raft_consensus.cc:493] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:43.745281 19125 raft_consensus.cc:515] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.746913 18778 leader_election.cc:290] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:43.747226 19125 leader_election.cc:290] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:43.749313 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431d298f513d47268284d50af2ea1436" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:43.752313 17753 leader_election.cc:343] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:43.756848 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 7ba365c977984e68a316356465b1e36a (DEFAULT_TABLE table=default.Table_05427967e4cc480cba4d8f7128b23677 [id=11bd6526602f4c0dbdb90bbc8806262b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:43.757890 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba365c977984e68a316356465b1e36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:43.761797 17799 tablet_service.cc:1511] Processing CreateTablet for tablet 2a1ac7e1b192465696640c02898bc39f (DEFAULT_TABLE table=default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:43.762761 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1ac7e1b192465696640c02898bc39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:43.753468 18231 ts_tablet_manager.cc:1929] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:43.763629 18231 log.cc:1199] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/f90a677e94034c73ab5452554862c1c3
I20260924 19:57:43.764465 18231 ts_tablet_manager.cc:1950] T f90a677e94034c73ab5452554862c1c3 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:43.766475 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 191) took 1700 ms. Trace:
I20260924 19:57:43.766845 18231 rpcz_store.cc:276] 0924 19:57:42.066169 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.078087 (+ 11918us) service_pool.cc:224] Handling call
0924 19:57:43.624445 (+1546358us) ts_tablet_manager.cc:1151] Deleting tablet f90a677e94034c73ab5452554862c1c3
0924 19:57:43.624481 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.670743 (+ 46262us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.753335 (+ 82592us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.766409 (+ 13074us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1486202}
I20260924 19:57:43.767314 18951 raft_consensus.cc:493] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:43.768170 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 195) took 1597 ms. Trace:
I20260924 19:57:43.767791 18951 raft_consensus.cc:515] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.768548 18231 rpcz_store.cc:276] 0924 19:57:42.170580 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.170693 (+ 113us) service_pool.cc:224] Handling call
0924 19:57:43.767938 (+1597245us) ts_tablet_manager.cc:1151] Deleting tablet 24f84032bca24a7ca9da65f0bbceb0d5
0924 19:57:43.767976 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.768125 (+ 149us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587743}
I20260924 19:57:43.769601 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 194) took 1599 ms. Trace:
I20260924 19:57:43.770957 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet f90a677e94034c73ab5452554862c1c3 (table default.Table_fca275d61c914e72b792ca0bb116d0bf [id=19596b2bff2e46399e96dcc6d5fbfbed]) successfully deleted
W20260924 19:57:43.771734 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 24f84032bca24a7ca9da65f0bbceb0d5 because the tablet was not found. No further retry: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5
I20260924 19:57:43.773089 18910 ts_tablet_manager.cc:1929] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260924 19:57:43.773128 18951 leader_election.cc:290] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:43.773470 18910 log.cc:1199] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/83ee43b6dc064dc7b320691d52e57ea7
I20260924 19:57:43.769933 18231 rpcz_store.cc:276] 0924 19:57:42.170225 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.170701 (+ 476us) service_pool.cc:224] Handling call
0924 19:57:43.769364 (+1598663us) ts_tablet_manager.cc:1151] Deleting tablet 349c6e9413bb41268d0c41d549203e2a
0924 19:57:43.769398 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.769556 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1526501}
W20260924 19:57:43.775561 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 349c6e9413bb41268d0c41d549203e2a because the tablet was not found. No further retry: Not found: Tablet not found: 349c6e9413bb41268d0c41d549203e2a
I20260924 19:57:43.777514 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 196) took 1584 ms. Trace:
I20260924 19:57:43.778098 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274f3decdc7e42c0917bf28151c4da71" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:43.777968 18231 rpcz_store.cc:276] 0924 19:57:42.193617 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.194109 (+ 492us) service_pool.cc:224] Handling call
0924 19:57:43.777263 (+1583154us) ts_tablet_manager.cc:1151] Deleting tablet 41bdb846ca19425b882434df0167479e
0924 19:57:43.777304 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.777466 (+ 162us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1527391}
I20260924 19:57:43.778743 17816 raft_consensus.cc:2468] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
W20260924 19:57:43.780519 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 41bdb846ca19425b882434df0167479e because the tablet was not found. No further retry: Not found: Tablet not found: 41bdb846ca19425b882434df0167479e
I20260924 19:57:43.782989 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 198) took 1578 ms. Trace:
I20260924 19:57:43.783361 18231 rpcz_store.cc:276] 0924 19:57:42.204356 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.204461 (+ 105us) service_pool.cc:224] Handling call
0924 19:57:43.782323 (+1577862us) ts_tablet_manager.cc:1151] Deleting tablet 2cf62a73c49245198d65011875148dc3
0924 19:57:43.782359 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.782514 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1532329}
I20260924 19:57:43.783805 18051 leader_election.cc:304] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
W20260924 19:57:43.784557 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 2cf62a73c49245198d65011875148dc3 because the tablet was not found. No further retry: Not found: Tablet not found: 2cf62a73c49245198d65011875148dc3
I20260924 19:57:43.784636 19080 raft_consensus.cc:2804] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:43.785081 19080 raft_consensus.cc:493] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:43.785401 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 197) took 1590 ms. Trace:
I20260924 19:57:43.785454 19080 raft_consensus.cc:3060] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:43.785766 18231 rpcz_store.cc:276] 0924 19:57:42.195290 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.195394 (+ 104us) service_pool.cc:224] Handling call
0924 19:57:43.785166 (+1589772us) ts_tablet_manager.cc:1151] Deleting tablet c884f12860164f12bbcd0d2f87067013
0924 19:57:43.785202 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.785357 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":183,"tablet-delete.queue_time_us":1535268}
W20260924 19:57:43.786998 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet c884f12860164f12bbcd0d2f87067013 because the tablet was not found. No further retry: Not found: Tablet not found: c884f12860164f12bbcd0d2f87067013
I20260924 19:57:43.788789 18231 tablet_replica.cc:333] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:43.789935 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274f3decdc7e42c0917bf28151c4da71" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:43.790320 18231 raft_consensus.cc:2243] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:43.790860 18231 raft_consensus.cc:2272] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:43.791268 18050 consensus_peers.cc:597] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:43.792758 18231 ts_tablet_manager.cc:1916] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:43.793622 18050 leader_election.cc:343] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:43.794699 18224 tablet_bootstrap.cc:654] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:43.779438 18220 tablet_bootstrap.cc:654] T affce4de8e0b412aa3b4bf19f9be7b6e P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:43.804358 17798 tablet_service.cc:1511] Processing CreateTablet for tablet accddc7f696142bfa6a5e655f5abc220 (DEFAULT_TABLE table=default.Table_bdae18e3011a421d96cdcc107e8f1dc2 [id=ca7affed79d94a2597fdaa1ff38e5e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:43.805403 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet accddc7f696142bfa6a5e655f5abc220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:43.805517 18910 ts_tablet_manager.cc:1950] T 83ee43b6dc064dc7b320691d52e57ea7 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:43.828224 17947 tablet_service.cc:1511] Processing CreateTablet for tablet 7ba365c977984e68a316356465b1e36a (DEFAULT_TABLE table=default.Table_05427967e4cc480cba4d8f7128b23677 [id=11bd6526602f4c0dbdb90bbc8806262b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:43.829355 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba365c977984e68a316356465b1e36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:43.833783 18231 ts_tablet_manager.cc:1929] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:43.845042 18090 tablet_service.cc:1511] Processing CreateTablet for tablet 7ba365c977984e68a316356465b1e36a (DEFAULT_TABLE table=default.Table_05427967e4cc480cba4d8f7128b23677 [id=11bd6526602f4c0dbdb90bbc8806262b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:43.848052 17606 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_a7d63063e7ee4493abe8c76da6d4f99d [id=be19c62729a34ae1a15966662303a869]). Expected version 2 got 1
I20260924 19:57:43.850678 17945 tablet_service.cc:1511] Processing CreateTablet for tablet 2a1ac7e1b192465696640c02898bc39f (DEFAULT_TABLE table=default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:43.851953 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1ac7e1b192465696640c02898bc39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:43.851727 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 180) took 2906 ms. Trace:
I20260924 19:57:43.834658 19125 raft_consensus.cc:493] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:43.859112 19125 raft_consensus.cc:515] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.861455 19125 leader_election.cc:290] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:43.862459 18231 log.cc:1199] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/910d61b8c3574302b5a328187226571e
I20260924 19:57:43.863499 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3192f66a492442c1bf4137cfe7e13bc9" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:43.864729 18231 ts_tablet_manager.cc:1950] T 910d61b8c3574302b5a328187226571e P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:43.867048 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 192) took 1746 ms. Trace:
I20260924 19:57:43.859633 18910 rpcz_store.cc:276] 0924 19:57:40.945056 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:40.945222 (+ 166us) service_pool.cc:224] Handling call
0924 19:57:43.240610 (+2295388us) ts_tablet_manager.cc:1151] Deleting tablet 83ee43b6dc064dc7b320691d52e57ea7
0924 19:57:43.240662 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.510167 (+269505us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.772956 (+262789us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.850570 (+ 77614us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2294805}
W20260924 19:57:43.880335 17753 leader_election.cc:343] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:43.883095 18231 rpcz_store.cc:276] 0924 19:57:42.120404 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.189808 (+ 69404us) service_pool.cc:224] Handling call
0924 19:57:43.788656 (+1598848us) ts_tablet_manager.cc:1151] Deleting tablet 910d61b8c3574302b5a328187226571e
0924 19:57:43.788695 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.802771 (+ 14076us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.833666 (+ 30895us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.866971 (+ 33305us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":731,"tablet-delete.queue_time_us":1538457}
I20260924 19:57:43.886621 18231 tablet_replica.cc:333] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:43.886893 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 910d61b8c3574302b5a328187226571e (table default.Table_2b157ac766d244e081168b817296bd4b [id=1bef6766d6a049a9a5126339c3f67842]) successfully deleted
I20260924 19:57:43.887817 18231 raft_consensus.cc:2243] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:43.893457 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 2a1ac7e1b192465696640c02898bc39f (DEFAULT_TABLE table=default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:43.897212 18231 raft_consensus.cc:2272] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:43.901018 17940 tablet_service.cc:1511] Processing CreateTablet for tablet accddc7f696142bfa6a5e655f5abc220 (DEFAULT_TABLE table=default.Table_bdae18e3011a421d96cdcc107e8f1dc2 [id=ca7affed79d94a2597fdaa1ff38e5e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:43.902228 17940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet accddc7f696142bfa6a5e655f5abc220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:43.904623 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1ac7e1b192465696640c02898bc39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:43.905514 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3192f66a492442c1bf4137cfe7e13bc9" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:43.906925 18965 raft_consensus.cc:515] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:43.908350 17905 leader_election.cc:343] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: d970cc92a1134a98a613d198a319cb01
I20260924 19:57:43.909158 17905 leader_election.cc:304] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:43.911351 19076 raft_consensus.cc:2749] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:43.920506 18231 ts_tablet_manager.cc:1916] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:43.933182 17753 leader_election.cc:343] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:43.933617 17753 leader_election.cc:304] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:43.935761 18780 raft_consensus.cc:2749] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:43.934862 17606 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:43.936527 18910 tablet_replica.cc:333] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:43.937438 18910 raft_consensus.cc:2243] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:43.939944 17798 tablet_service.cc:1511] Processing CreateTablet for tablet 7c1afbfc7de7480ab4be40680a3e3ada (DEFAULT_TABLE table=default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:43.940869 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c1afbfc7de7480ab4be40680a3e3ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:43.909669 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba365c977984e68a316356465b1e36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:43.945261 19051 tablet_replica.cc:333] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:43.935890 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 83ee43b6dc064dc7b320691d52e57ea7 (table default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]) successfully deleted
I20260924 19:57:43.955854 17606 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:43.977237 19051 raft_consensus.cc:2243] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:43.977941 19051 raft_consensus.cc:2272] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:43.911011 19080 raft_consensus.cc:515] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:43.928633 18095 tablet_service.cc:1511] Processing CreateTablet for tablet accddc7f696142bfa6a5e655f5abc220 (DEFAULT_TABLE table=default.Table_bdae18e3011a421d96cdcc107e8f1dc2 [id=ca7affed79d94a2597fdaa1ff38e5e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:43.979239 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet accddc7f696142bfa6a5e655f5abc220. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:43.910382 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431d298f513d47268284d50af2ea1436" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:43.986882 18231 ts_tablet_manager.cc:1929] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:43.987274 18231 log.cc:1199] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/07f86f24d9d14d05898e1c0f69428850
I20260924 19:57:43.987955 18231 ts_tablet_manager.cc:1950] T 07f86f24d9d14d05898e1c0f69428850 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:43.988688 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 199) took 1737 ms. Trace:
I20260924 19:57:43.988976 18231 rpcz_store.cc:276] 0924 19:57:42.251284 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.261647 (+ 10363us) service_pool.cc:224] Handling call
0924 19:57:43.886475 (+1624828us) ts_tablet_manager.cc:1151] Deleting tablet 07f86f24d9d14d05898e1c0f69428850
0924 19:57:43.886517 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:43.962595 (+ 76078us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.986751 (+ 24156us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:43.988637 (+ 1886us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1622673}
W20260924 19:57:43.990087 17753 leader_election.cc:343] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:43.990264 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 07f86f24d9d14d05898e1c0f69428850 (table default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]) successfully deleted
I20260924 19:57:43.990568 17753 leader_election.cc:304] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:43.991418 18780 raft_consensus.cc:2749] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:44.015408 18231 tablet_replica.cc:333] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:44.018060 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 7c1afbfc7de7480ab4be40680a3e3ada (DEFAULT_TABLE table=default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.018507 18231 raft_consensus.cc:2243] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:44.022603 17611 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_a57608cee13f4af2b03aa54baa4acb7f" } modify_external_catalogs: true
I20260924 19:57:44.025182 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c1afbfc7de7480ab4be40680a3e3ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.025527 18965 leader_election.cc:290] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:44.035432 17611 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_a57608cee13f4af2b03aa54baa4acb7f" } modify_external_catalogs: true
I20260924 19:57:44.036441 17967 raft_consensus.cc:3060] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:44.040154 18780 raft_consensus.cc:493] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:44.040645 18780 raft_consensus.cc:515] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.018985 18231 raft_consensus.cc:2272] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:44.042531 18780 leader_election.cc:290] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:44.044368 19080 leader_election.cc:290] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:44.045683 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92c0daa7d67e4aa390e334c669adda76" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:44.046197 17816 raft_consensus.cc:3060] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:44.054400 17816 raft_consensus.cc:2468] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:44.057498 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "484517ee7df8463bb70d201ead1c6835" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:44.059993 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274f3decdc7e42c0917bf28151c4da71" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:44.060956 17816 raft_consensus.cc:3060] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:44.063907 18051 leader_election.cc:304] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:44.064656 18951 consensus_queue.cc:1048] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:44.065853 18951 raft_consensus.cc:2804] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:44.066633 17606 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
I20260924 19:57:44.066476 18951 raft_consensus.cc:697] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:44.068100 18951 consensus_queue.cc:237] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.073690 18231 ts_tablet_manager.cc:1916] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:44.084605 18051 consensus_peers.cc:597] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07f86f24d9d14d05898e1c0f69428850. This is attempt 1: this message will repeat every 5th retry.
W20260924 19:57:44.086508 17753 leader_election.cc:343] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:44.092610 17816 raft_consensus.cc:2468] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:44.099298 17607 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 3192f66a492442c1bf4137cfe7e13bc9
I20260924 19:57:44.100777 17607 catalog_manager.cc:6934] ReplaceTablet: tablet 3192f66a492442c1bf4137cfe7e13bc9 deleted and replaced by tablet 144355b18c2f413baa28dd632623db52
I20260924 19:57:44.101743 17607 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_4df831f35ae74b2f86e7e1b653b69867" } new_table_name: "default.Table_7c27cdce0a3b40f69142de52aaf39493" modify_external_catalogs: true
I20260924 19:57:44.102665 17611 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet bf98ebc764004bc59a764f42edcc759e
I20260924 19:57:44.103740 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_4c4cbf180cb04cf78a5d09a34467e15a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:44.102303 17798 tablet_service.cc:1558] Processing DeleteTablet for tablet 3192f66a492442c1bf4137cfe7e13bc9 with delete_type TABLET_DATA_DELETED (replaced by tablet 144355b18c2f413baa28dd632623db52) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:44.105448 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c4cbf180cb04cf78a5d09a34467e15a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:44.106051 18051 leader_election.cc:304] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:44.106967 18965 raft_consensus.cc:2804] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:44.108906 18965 raft_consensus.cc:697] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:44.109791 18965 consensus_queue.cc:237] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.119346 19051 ts_tablet_manager.cc:1916] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:44.114295 18223 tablet_bootstrap.cc:700] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent opening tablet: real 0.131s user 0.004s sys 0.000s
I20260924 19:57:44.128445 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 3192f66a492442c1bf4137cfe7e13bc9 with delete_type TABLET_DATA_DELETED (replaced by tablet 144355b18c2f413baa28dd632623db52) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:44.037256 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92c0daa7d67e4aa390e334c669adda76" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
W20260924 19:57:44.139632 18050 leader_election.cc:343] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:44.123636 18223 tablet_bootstrap.cc:654] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.146333 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 8c05ec2ba01448d998c6ffa1525de984 (DEFAULT_TABLE table=default.Table_14bf9267f2da4ba48354027fc92cab3d [id=b898f46c59b246e39fe8b62a45019ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.147506 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c05ec2ba01448d998c6ffa1525de984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.148147 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274f3decdc7e42c0917bf28151c4da71" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:44.148542 17940 tablet_service.cc:1511] Processing CreateTablet for tablet 7c1afbfc7de7480ab4be40680a3e3ada (DEFAULT_TABLE table=default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:57:44.151749 18050 leader_election.cc:343] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:44.157126 17940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c1afbfc7de7480ab4be40680a3e3ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.162240 17798 tablet_service.cc:1511] Processing CreateTablet for tablet c058a49bfd5544b58b76edfd3fadbd1a (DEFAULT_TABLE table=default.Table_ee93a1803fd54a4ba6723d1e3783c004 [id=fec7f8aa891e42c1a84fe32fe4e0c44f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.163606 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c058a49bfd5544b58b76edfd3fadbd1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.180157 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 8c05ec2ba01448d998c6ffa1525de984 (DEFAULT_TABLE table=default.Table_14bf9267f2da4ba48354027fc92cab3d [id=b898f46c59b246e39fe8b62a45019ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.195996 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c05ec2ba01448d998c6ffa1525de984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.179301 18910 raft_consensus.cc:2272] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:44.211185 18050 consensus_peers.cc:597] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:44.212791 18224 tablet_bootstrap.cc:492] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:44.213212 18224 ts_tablet_manager.cc:1403] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.491s user 0.036s sys 0.061s
I20260924 19:57:44.215993 18224 raft_consensus.cc:359] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.216598 18224 raft_consensus.cc:385] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:57:44.217406 18050 consensus_peers.cc:597] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:44.219274 18220 tablet_bootstrap.cc:492] T affce4de8e0b412aa3b4bf19f9be7b6e P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:44.219692 18220 ts_tablet_manager.cc:1403] T affce4de8e0b412aa3b4bf19f9be7b6e P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.599s user 0.011s sys 0.016s
I20260924 19:57:44.221619 18910 ts_tablet_manager.cc:1916] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:44.222568 18220 raft_consensus.cc:359] T affce4de8e0b412aa3b4bf19f9be7b6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:44.224118 17945 tablet_service.cc:1511] Processing CreateTablet for tablet 8c05ec2ba01448d998c6ffa1525de984 (DEFAULT_TABLE table=default.Table_14bf9267f2da4ba48354027fc92cab3d [id=b898f46c59b246e39fe8b62a45019ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.225175 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c05ec2ba01448d998c6ffa1525de984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.242077 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "484517ee7df8463bb70d201ead1c6835" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:44.256014 18084 tablet_service.cc:1511] Processing CreateTablet for tablet c058a49bfd5544b58b76edfd3fadbd1a (DEFAULT_TABLE table=default.Table_ee93a1803fd54a4ba6723d1e3783c004 [id=fec7f8aa891e42c1a84fe32fe4e0c44f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.256846 17947 tablet_service.cc:1558] Processing DeleteTablet for tablet 3192f66a492442c1bf4137cfe7e13bc9 with delete_type TABLET_DATA_DELETED (replaced by tablet 144355b18c2f413baa28dd632623db52) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:44.262813 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c058a49bfd5544b58b76edfd3fadbd1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.265153 17947 tablet_service.cc:1511] Processing CreateTablet for tablet c058a49bfd5544b58b76edfd3fadbd1a (DEFAULT_TABLE table=default.Table_ee93a1803fd54a4ba6723d1e3783c004 [id=fec7f8aa891e42c1a84fe32fe4e0c44f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.266013 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c058a49bfd5544b58b76edfd3fadbd1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.268568 19080 consensus_queue.cc:1048] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:44.216835 18224 raft_consensus.cc:740] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.286895 18224 consensus_queue.cc:260] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.288887 18224 ts_tablet_manager.cc:1434] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.075s user 0.006s sys 0.000s
I20260924 19:57:44.289710 18224 tablet_bootstrap.cc:492] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
W20260924 19:57:44.268421 17753 leader_election.cc:343] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:44.291527 17753 leader_election.cc:304] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:44.292168 18780 raft_consensus.cc:2749] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:44.294502 18220 raft_consensus.cc:385] T affce4de8e0b412aa3b4bf19f9be7b6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.294798 18220 raft_consensus.cc:740] T affce4de8e0b412aa3b4bf19f9be7b6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.295432 18220 consensus_queue.cc:260] T affce4de8e0b412aa3b4bf19f9be7b6e P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:44.296650 19051 ts_tablet_manager.cc:1929] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:44.297649 18220 ts_tablet_manager.cc:1434] T affce4de8e0b412aa3b4bf19f9be7b6e P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.078s user 0.004s sys 0.000s
I20260924 19:57:44.298558 18220 tablet_bootstrap.cc:492] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:44.298583 19051 log.cc:1199] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/a765abd702644f079bddffb3fc4238ef
I20260924 19:57:44.305100 19051 ts_tablet_manager.cc:1950] T a765abd702644f079bddffb3fc4238ef P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:44.310652 18223 tablet_bootstrap.cc:492] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:44.311069 18223 ts_tablet_manager.cc:1403] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.614s user 0.010s sys 0.001s
I20260924 19:57:44.321887 18223 raft_consensus.cc:359] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.322609 18223 raft_consensus.cc:385] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.322886 18223 raft_consensus.cc:740] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.330271 18223 consensus_queue.cc:260] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.331233 19076 raft_consensus.cc:493] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eaf64bfb64aa42948c9bc1411ce4ce35)
I20260924 19:57:44.331740 19076 raft_consensus.cc:515] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.332412 18223 ts_tablet_manager.cc:1434] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260924 19:57:44.334098 19076 leader_election.cc:290] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:44.335464 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0246d83189d4ba5b75fbd8390f517f6" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:44.345479 17612 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet dd434e38c9364f55bd18383149015028 (table default.Table_a7d63063e7ee4493abe8c76da6d4f99d [id=be19c62729a34ae1a15966662303a869]). Expected version 2 got 1
I20260924 19:57:44.346336 17612 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:44.347178 17612 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:44.381018 18910 ts_tablet_manager.cc:1929] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:44.381433 18910 log.cc:1199] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/f90a677e94034c73ab5452554862c1c3
I20260924 19:57:44.405236 17612 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:44.405828 18166 heartbeater.cc:344] Connected to a master server at 127.17.2.190:46309
I20260924 19:57:44.407203 18910 ts_tablet_manager.cc:1950] T f90a677e94034c73ab5452554862c1c3 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:44.411228 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 188) took 2340 ms. Trace:
I20260924 19:57:44.411569 18910 rpcz_store.cc:276] 0924 19:57:42.071138 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.071266 (+ 128us) service_pool.cc:224] Handling call
0924 19:57:43.936334 (+1865068us) ts_tablet_manager.cc:1151] Deleting tablet f90a677e94034c73ab5452554862c1c3
0924 19:57:43.936391 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.246108 (+309717us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.380765 (+134657us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.411127 (+ 30362us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1851950}
I20260924 19:57:44.413045 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet f90a677e94034c73ab5452554862c1c3 (table default.Table_fca275d61c914e72b792ca0bb116d0bf [id=19596b2bff2e46399e96dcc6d5fbfbed]) successfully deleted
I20260924 19:57:44.406657 17612 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
I20260924 19:57:44.424768 17612 catalog_manager.cc:5838] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:44.425419 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 200) took 2211 ms. Trace:
I20260924 19:57:44.425637 18223 tablet_bootstrap.cc:492] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:44.425787 19051 rpcz_store.cc:276] 0924 19:57:42.214237 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.214348 (+ 111us) service_pool.cc:224] Handling call
0924 19:57:43.896551 (+1682203us) ts_tablet_manager.cc:1151] Deleting tablet a765abd702644f079bddffb3fc4238ef
0924 19:57:43.945066 (+ 48515us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b5800000f98. stack: 00007fb83c60c323 00007fb83c60c220 00007fb83c1b0574 0000000000516964 00000000005168f9 00007fb83c4c28a1 00007fb83c606b51 00007fb83c607182 00007fb845578ec9 00007fb84478b918 00007fb8448522c9 00007fb8448528c3 00007fb844883960 00007fb844869e6d 00007fb844869e19 00007fb844869da9
0924 19:57:43.945120 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.201647 (+256527us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.289859 (+ 88212us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.425347 (+135488us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5123,"spinlock_wait_cycles":25620224,"tablet-delete.queue_time_us":1664752}
I20260924 19:57:44.428414 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 201) took 1522 ms. Trace:
I20260924 19:57:44.428778 19051 rpcz_store.cc:276] 0924 19:57:42.913165 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.913839 (+ 674us) service_pool.cc:224] Handling call
0924 19:57:44.428025 (+1514186us) ts_tablet_manager.cc:1151] Deleting tablet ae7a0660795e4e4b87d0959ca84d5d1b
0924 19:57:44.428071 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.428350 (+ 279us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1467831}
I20260924 19:57:44.430775 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 190) took 2298 ms. Trace:
I20260924 19:57:44.430846 19051 tablet_replica.cc:333] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:44.429735 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet a765abd702644f079bddffb3fc4238ef (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]) successfully deleted
W20260924 19:57:44.431715 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet ae7a0660795e4e4b87d0959ca84d5d1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae7a0660795e4e4b87d0959ca84d5d1b already in progress: creating tablet
I20260924 19:57:44.432416 18223 tablet_bootstrap.cc:654] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.432708 17612 catalog_manager.cc:5838] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:44.438186 18224 tablet_bootstrap.cc:654] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.439049 19051 raft_consensus.cc:2243] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:44.439586 19051 raft_consensus.cc:2272] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:44.441336 19051 ts_tablet_manager.cc:1916] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:44.446271 18223 tablet_bootstrap.cc:492] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:44.446688 18223 ts_tablet_manager.cc:1403] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.000s
I20260924 19:57:44.449118 18223 raft_consensus.cc:359] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.449689 18223 raft_consensus.cc:385] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.449913 18223 raft_consensus.cc:740] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.450524 18223 consensus_queue.cc:260] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.451776 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e0246d83189d4ba5b75fbd8390f517f6" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:44.456799 18223 ts_tablet_manager.cc:1434] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260924 19:57:44.457684 18223 tablet_bootstrap.cc:492] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:44.465745 18223 tablet_bootstrap.cc:654] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.476284 18223 tablet_bootstrap.cc:492] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:44.477273 18223 ts_tablet_manager.cc:1403] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.011s
I20260924 19:57:44.479956 18223 raft_consensus.cc:359] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:44.480628 18223 raft_consensus.cc:385] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.480921 18223 raft_consensus.cc:740] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.481627 18223 consensus_queue.cc:260] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:44.484303 18223 ts_tablet_manager.cc:1434] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260924 19:57:44.485948 18223 tablet_bootstrap.cc:492] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:44.453243 17906 leader_election.cc:304] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:44.491709 18223 tablet_bootstrap.cc:654] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:44.492328 17906 consensus_peers.cc:597] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 910d61b8c3574302b5a328187226571e. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:44.494957 19076 raft_consensus.cc:2749] T e0246d83189d4ba5b75fbd8390f517f6 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:44.502250 18223 tablet_bootstrap.cc:492] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:44.502656 18223 ts_tablet_manager.cc:1403] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.002s
I20260924 19:57:44.491019 18910 rpcz_store.cc:276] 0924 19:57:42.132028 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.136851 (+ 4823us) service_pool.cc:224] Handling call
0924 19:57:44.429713 (+2292862us) ts_tablet_manager.cc:1151] Deleting tablet c884f12860164f12bbcd0d2f87067013
0924 19:57:44.430108 (+ 395us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.430708 (+ 600us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2292172}
I20260924 19:57:44.504454 18910 tablet_replica.cc:333] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
W20260924 19:57:44.505045 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet c884f12860164f12bbcd0d2f87067013 because the tablet was not found. No further retry: Not found: Tablet not found: c884f12860164f12bbcd0d2f87067013
I20260924 19:57:44.505174 18910 raft_consensus.cc:2243] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:44.505160 18223 raft_consensus.cc:359] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.505764 18223 raft_consensus.cc:385] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.506011 18223 raft_consensus.cc:740] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.506083 18910 raft_consensus.cc:2272] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:44.506814 18223 consensus_queue.cc:260] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.509449 18223 ts_tablet_manager.cc:1434] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260924 19:57:44.510452 18223 tablet_bootstrap.cc:492] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:44.410032 19025 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:44.526461 18223 tablet_bootstrap.cc:654] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.532313 18224 tablet_bootstrap.cc:492] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:44.532720 18224 ts_tablet_manager.cc:1403] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.243s user 0.010s sys 0.003s
I20260924 19:57:44.457707 18220 tablet_bootstrap.cc:654] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.522893 18910 ts_tablet_manager.cc:1916] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:44.534159 18231 ts_tablet_manager.cc:1929] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:44.534626 18231 log.cc:1199] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/ae7a0660795e4e4b87d0959ca84d5d1b
I20260924 19:57:44.535382 18231 ts_tablet_manager.cc:1950] T ae7a0660795e4e4b87d0959ca84d5d1b P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:44.552645 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 200) took 1952 ms. Trace:
I20260924 19:57:44.544706 18224 raft_consensus.cc:359] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.560730 18224 raft_consensus.cc:385] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.561069 18224 raft_consensus.cc:740] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.559291 18231 rpcz_store.cc:276] 0924 19:57:42.599848 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.599961 (+ 113us) service_pool.cc:224] Handling call
0924 19:57:44.001087 (+1401126us) ts_tablet_manager.cc:1151] Deleting tablet ae7a0660795e4e4b87d0959ca84d5d1b
0924 19:57:44.001941 (+ 854us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.488207 (+486266us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.534027 (+ 45820us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.552573 (+ 18546us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1400465}
I20260924 19:57:44.563949 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet ae7a0660795e4e4b87d0959ca84d5d1b (table default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df]) successfully deleted
I20260924 19:57:44.564894 18231 tablet_replica.cc:333] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:44.561820 18224 consensus_queue.cc:260] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.566216 18224 ts_tablet_manager.cc:1434] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.033s user 0.004s sys 0.002s
I20260924 19:57:44.568334 18223 tablet_bootstrap.cc:492] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:44.568753 18223 ts_tablet_manager.cc:1403] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.005s
I20260924 19:57:44.570393 18224 tablet_bootstrap.cc:492] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:44.571074 18223 raft_consensus.cc:359] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.571754 18223 raft_consensus.cc:385] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.572013 18223 raft_consensus.cc:740] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
W20260924 19:57:44.575377 17601 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet d930e7f4a6b7471899dab1b28e40f93d (table default.Table_9b8f6ad547fc4c3fa379c93590aebd56 [id=4fc8e565a6a640549716cee596fa3f9d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d930e7f4a6b7471899dab1b28e40f93d consensus state has no leader
I20260924 19:57:44.583940 18224 tablet_bootstrap.cc:654] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.590497 18910 ts_tablet_manager.cc:1929] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:44.593042 18231 raft_consensus.cc:2243] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:44.594631 18231 raft_consensus.cc:2272] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:44.596403 18231 ts_tablet_manager.cc:1916] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:44.597239 18223 consensus_queue.cc:260] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.599627 18223 ts_tablet_manager.cc:1434] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20260924 19:57:44.600600 18223 tablet_bootstrap.cc:492] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:44.612854 18223 tablet_bootstrap.cc:654] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.615017 18910 log.cc:1199] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/a765abd702644f079bddffb3fc4238ef
I20260924 19:57:44.615984 18910 ts_tablet_manager.cc:1950] T a765abd702644f079bddffb3fc4238ef P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:44.618052 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 192) took 2464 ms. Trace:
I20260924 19:57:44.623920 18910 rpcz_store.cc:276] 0924 19:57:42.153924 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.154020 (+ 96us) service_pool.cc:224] Handling call
0924 19:57:44.504296 (+2350276us) ts_tablet_manager.cc:1151] Deleting tablet a765abd702644f079bddffb3fc4238ef
0924 19:57:44.504342 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.542140 (+ 37798us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.590374 (+ 48234us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.617992 (+ 27618us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2349716}
I20260924 19:57:44.632481 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 191) took 2478 ms. Trace:
I20260924 19:57:44.633937 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet a765abd702644f079bddffb3fc4238ef (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]) successfully deleted
I20260924 19:57:44.634352 18224 tablet_bootstrap.cc:492] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:44.634848 18224 ts_tablet_manager.cc:1403] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.000s
I20260924 19:57:44.633615 19025 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e059bbe4dac4178bccb3be48cb1118d, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.638324 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 5f116f866c2b4674b4eef531cf00af66 (DEFAULT_TABLE table=default.Table_6ddb9314aa114575b5317b1d08ba138e [id=87fa4ac369bf465ca534359e522d3cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.639596 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f116f866c2b4674b4eef531cf00af66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.647796 18780 raft_consensus.cc:493] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:44.648314 18780 raft_consensus.cc:515] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.649371 18084 tablet_service.cc:1511] Processing CreateTablet for tablet 5f116f866c2b4674b4eef531cf00af66 (DEFAULT_TABLE table=default.Table_6ddb9314aa114575b5317b1d08ba138e [id=87fa4ac369bf465ca534359e522d3cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.650372 18780 leader_election.cc:290] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:44.651384 18224 raft_consensus.cc:359] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:44.652069 18224 raft_consensus.cc:385] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.652335 18224 raft_consensus.cc:740] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.651245 17945 tablet_service.cc:1511] Processing CreateTablet for tablet 5f116f866c2b4674b4eef531cf00af66 (DEFAULT_TABLE table=default.Table_6ddb9314aa114575b5317b1d08ba138e [id=87fa4ac369bf465ca534359e522d3cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.652907 18910 rpcz_store.cc:276] 0924 19:57:42.153613 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.154660 (+ 1047us) service_pool.cc:224] Handling call
0924 19:57:44.632158 (+2477498us) ts_tablet_manager.cc:1151] Deleting tablet 24f84032bca24a7ca9da65f0bbceb0d5
0924 19:57:44.632221 (+ 63us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.632424 (+ 203us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2476890}
I20260924 19:57:44.653558 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f116f866c2b4674b4eef531cf00af66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.653693 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924bc897f48b44069f1fb2ee71e26db7" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:44.654100 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 193) took 2499 ms. Trace:
I20260924 19:57:44.652436 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924bc897f48b44069f1fb2ee71e26db7" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:44.654440 18910 rpcz_store.cc:276] 0924 19:57:42.154976 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.155317 (+ 341us) service_pool.cc:224] Handling call
0924 19:57:44.653838 (+2498521us) ts_tablet_manager.cc:1151] Deleting tablet 41bdb846ca19425b882434df0167479e
0924 19:57:44.653881 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.654041 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2497986}
I20260924 19:57:44.655537 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 194) took 2500 ms. Trace:
W20260924 19:57:44.655828 17753 leader_election.cc:343] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:44.655841 18910 rpcz_store.cc:276] 0924 19:57:42.155356 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.159295 (+ 3939us) service_pool.cc:224] Handling call
0924 19:57:44.655308 (+2496013us) ts_tablet_manager.cc:1151] Deleting tablet 2cf62a73c49245198d65011875148dc3
0924 19:57:44.655350 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.655489 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2495442}
W20260924 19:57:44.656436 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 24f84032bca24a7ca9da65f0bbceb0d5 because the tablet was not found. No further retry: Not found: Tablet not found: 24f84032bca24a7ca9da65f0bbceb0d5
W20260924 19:57:44.656515 17753 leader_election.cc:343] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:44.656790 18910 tablet_replica.cc:333] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:44.656876 17753 leader_election.cc:304] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:44.657141 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 41bdb846ca19425b882434df0167479e because the tablet was not found. No further retry: Not found: Tablet not found: 41bdb846ca19425b882434df0167479e
I20260924 19:57:44.657472 18910 raft_consensus.cc:2243] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:44.658322 18910 raft_consensus.cc:2272] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:44.661026 18220 tablet_bootstrap.cc:492] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:44.661510 18220 ts_tablet_manager.cc:1403] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.363s user 0.004s sys 0.008s
W20260924 19:57:44.662047 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 2cf62a73c49245198d65011875148dc3 because the tablet was not found. No further retry: Not found: Tablet not found: 2cf62a73c49245198d65011875148dc3
I20260924 19:57:44.662683 18224 consensus_queue.cc:260] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:44.664490 18780 raft_consensus.cc:2749] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:44.667514 17798 tablet_service.cc:1511] Processing CreateTablet for tablet b7a1887d396f40a1a0c153dd0b2fe307 (DEFAULT_TABLE table=default.Table_954dba173a70455b90daee9230b44b6f [id=b743c3e3916a45069fabeed784e9e249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.667771 17947 tablet_service.cc:1511] Processing CreateTablet for tablet b7a1887d396f40a1a0c153dd0b2fe307 (DEFAULT_TABLE table=default.Table_954dba173a70455b90daee9230b44b6f [id=b743c3e3916a45069fabeed784e9e249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.668605 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7a1887d396f40a1a0c153dd0b2fe307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.668700 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7a1887d396f40a1a0c153dd0b2fe307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.672468 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f116f866c2b4674b4eef531cf00af66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.682101 18220 raft_consensus.cc:359] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.681694 18231 ts_tablet_manager.cc:1929] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:44.689024 18082 tablet_service.cc:1511] Processing CreateTablet for tablet b7a1887d396f40a1a0c153dd0b2fe307 (DEFAULT_TABLE table=default.Table_954dba173a70455b90daee9230b44b6f [id=b743c3e3916a45069fabeed784e9e249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.689738 18231 log.cc:1199] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:57:44.690500 18231 ts_tablet_manager.cc:1950] T ba46ac86b242407abd3e8dbf09f12efa P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:44.690990 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7a1887d396f40a1a0c153dd0b2fe307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.682837 18220 raft_consensus.cc:385] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.694828 17940 tablet_service.cc:1511] Processing CreateTablet for tablet 7635c52883ec4434bad0cc15f8733f72 (DEFAULT_TABLE table=default.Table_0673b53c69bf4c429697ba739ef113e4 [id=ea252b103a0f43d18971c78fe2174c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.695307 18220 raft_consensus.cc:740] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.696120 17940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7635c52883ec4434bad0cc15f8733f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.696003 18220 consensus_queue.cc:260] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.697348 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 201) took 1743 ms. Trace:
I20260924 19:57:44.704996 18220 ts_tablet_manager.cc:1434] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.043s user 0.006s sys 0.001s
I20260924 19:57:44.706056 18220 tablet_bootstrap.cc:492] T 92c0daa7d67e4aa390e334c669adda76 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:44.635926 19025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:44.710299 18231 rpcz_store.cc:276] 0924 19:57:42.954279 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.954438 (+ 159us) service_pool.cc:224] Handling call
0924 19:57:44.564743 (+1610305us) ts_tablet_manager.cc:1151] Deleting tablet ba46ac86b242407abd3e8dbf09f12efa
0924 19:57:44.564789 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.604808 (+ 40019us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.681575 (+ 76767us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.697289 (+ 15714us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26881,"tablet-delete.queue_time_us":1609748}
I20260924 19:57:44.714936 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 7635c52883ec4434bad0cc15f8733f72 (DEFAULT_TABLE table=default.Table_0673b53c69bf4c429697ba739ef113e4 [id=ea252b103a0f43d18971c78fe2174c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.715358 18910 ts_tablet_manager.cc:1916] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:44.716358 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7635c52883ec4434bad0cc15f8733f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.722343 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:44.724965 18231 tablet_replica.cc:333] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:44.725107 18223 tablet_bootstrap.cc:492] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:44.725646 18223 ts_tablet_manager.cc:1403] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.125s user 0.014s sys 0.006s
I20260924 19:57:44.726797 18231 raft_consensus.cc:2243] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:44.728523 18231 raft_consensus.cc:2272] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:44.735889 17950 tablet_service.cc:1511] Processing CreateTablet for tablet bf025253c49a4c23b79a25333cad160e (DEFAULT_TABLE table=default.Table_1d9dfc2ce03046ba955d3d82049402fd [id=c44dfe41598f4e6cbf320e3044b49fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.737025 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf025253c49a4c23b79a25333cad160e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.740705 18231 ts_tablet_manager.cc:1916] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:44.765992 18095 tablet_service.cc:1511] Processing CreateTablet for tablet bf025253c49a4c23b79a25333cad160e (DEFAULT_TABLE table=default.Table_1d9dfc2ce03046ba955d3d82049402fd [id=c44dfe41598f4e6cbf320e3044b49fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.767220 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf025253c49a4c23b79a25333cad160e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.723934 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet ba46ac86b242407abd3e8dbf09f12efa (table default.Table_6115a46b8e7444e6899712d31f41390a [id=6e83504d58ee4c52b83c9109bb7bfa1b]) successfully deleted
W20260924 19:57:44.779487 17600 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 55581us
I20260924 19:57:44.781811 17798 tablet_service.cc:1511] Processing CreateTablet for tablet 7635c52883ec4434bad0cc15f8733f72 (DEFAULT_TABLE table=default.Table_0673b53c69bf4c429697ba739ef113e4 [id=ea252b103a0f43d18971c78fe2174c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.783169 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7635c52883ec4434bad0cc15f8733f72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.785876 18223 raft_consensus.cc:359] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.786576 18223 raft_consensus.cc:385] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.786850 18223 raft_consensus.cc:740] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.788075 19151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:44.788903 19151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:44.793025 18224 ts_tablet_manager.cc:1434] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.158s user 0.013s sys 0.005s
I20260924 19:57:44.794059 18224 tablet_bootstrap.cc:492] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:44.798291 17800 tablet_service.cc:1511] Processing CreateTablet for tablet bf025253c49a4c23b79a25333cad160e (DEFAULT_TABLE table=default.Table_1d9dfc2ce03046ba955d3d82049402fd [id=c44dfe41598f4e6cbf320e3044b49fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.799446 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf025253c49a4c23b79a25333cad160e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.810822 19051 ts_tablet_manager.cc:1929] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:44.811275 19051 log.cc:1199] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:57:44.811968 19051 ts_tablet_manager.cc:1950] T ba46ac86b242407abd3e8dbf09f12efa P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:44.812464 18224 tablet_bootstrap.cc:654] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.815912 19025 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:57:44.819679 18910 ts_tablet_manager.cc:1929] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:44.820079 18910 log.cc:1199] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/910d61b8c3574302b5a328187226571e
I20260924 19:57:44.820830 18910 ts_tablet_manager.cc:1950] T 910d61b8c3574302b5a328187226571e P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:44.825748 17949 tablet_service.cc:1511] Processing CreateTablet for tablet cb111f7a189c4f00a41686d647c64e65 (DEFAULT_TABLE table=default.Table_8be4bd6989c649469efa4d12976b4576 [id=8bee13abe4024a1e8c1c1ef8d14e26ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.826829 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb111f7a189c4f00a41686d647c64e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.794704 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:44.829742 18090 tablet_service.cc:1511] Processing CreateTablet for tablet cb111f7a189c4f00a41686d647c64e65 (DEFAULT_TABLE table=default.Table_8be4bd6989c649469efa4d12976b4576 [id=8bee13abe4024a1e8c1c1ef8d14e26ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.830029 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
I20260924 19:57:44.830596 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 3 got 0
I20260924 19:57:44.831454 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb111f7a189c4f00a41686d647c64e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.832365 17799 tablet_service.cc:1511] Processing CreateTablet for tablet cb111f7a189c4f00a41686d647c64e65 (DEFAULT_TABLE table=default.Table_8be4bd6989c649469efa4d12976b4576 [id=8bee13abe4024a1e8c1c1ef8d14e26ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.834605 18223 consensus_queue.cc:260] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.836241 18937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 253 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:44.837010 18223 ts_tablet_manager.cc:1434] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.111s user 0.006s sys 0.000s
W20260924 19:57:44.837371 18937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55566 (request call id 1) took 2580 ms (client timeout 627 ms). Trace:
I20260924 19:57:44.837910 18223 tablet_bootstrap.cc:492] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:44.837769 18937 rpcz_store.cc:269] 0924 19:57:42.256609 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.256810 (+ 201us) service_pool.cc:224] Handling call
0924 19:57:44.837290 (+2580480us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:44.839526 18938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 253 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
W20260924 19:57:44.840324 18938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55566 (request call id 0) took 3229 ms (client timeout 627 ms). Trace:
W20260924 19:57:44.840680 18938 rpcz_store.cc:269] 0924 19:57:41.610858 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:41.611052 (+ 194us) service_pool.cc:224] Handling call
0924 19:57:44.840269 (+3229217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:44.852079 18231 ts_tablet_manager.cc:1929] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:44.852624 18231 log.cc:1199] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/44199d2655f14dcf97afe2180789aae4
I20260924 19:57:44.853176 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 204) took 1754 ms. Trace:
I20260924 19:57:44.853364 18231 ts_tablet_manager.cc:1950] T 44199d2655f14dcf97afe2180789aae4 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:44.853528 19051 rpcz_store.cc:276] 0924 19:57:43.098895 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:43.125751 (+ 26856us) service_pool.cc:224] Handling call
0924 19:57:44.430680 (+1304929us) ts_tablet_manager.cc:1151] Deleting tablet ba46ac86b242407abd3e8dbf09f12efa
0924 19:57:44.430730 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.599269 (+168539us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.810686 (+211417us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.853110 (+ 42424us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39064,"tablet-delete.queue_time_us":1303591}
I20260924 19:57:44.855021 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet ba46ac86b242407abd3e8dbf09f12efa (table default.Table_6115a46b8e7444e6899712d31f41390a [id=6e83504d58ee4c52b83c9109bb7bfa1b]) successfully deleted
I20260924 19:57:44.822975 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 189) took 2701 ms. Trace:
I20260924 19:57:44.863652 18910 rpcz_store.cc:276] 0924 19:57:42.121184 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.164330 (+ 43146us) service_pool.cc:224] Handling call
0924 19:57:44.656649 (+2492319us) ts_tablet_manager.cc:1151] Deleting tablet 910d61b8c3574302b5a328187226571e
0924 19:57:44.656686 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.745366 (+ 88680us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.819550 (+ 74184us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.822921 (+ 3371us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2491716}
I20260924 19:57:44.866458 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 910d61b8c3574302b5a328187226571e (table default.Table_2b157ac766d244e081168b817296bd4b [id=1bef6766d6a049a9a5126339c3f67842]) successfully deleted
I20260924 19:57:44.869415 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb111f7a189c4f00a41686d647c64e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.871081 18910 tablet_replica.cc:333] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:44.871822 18910 raft_consensus.cc:2243] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:44.873076 18910 raft_consensus.cc:2272] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:44.874755 18910 ts_tablet_manager.cc:1916] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:44.876266 18231 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 202) took 1905 ms. Trace:
I20260924 19:57:44.876606 18231 rpcz_store.cc:276] 0924 19:57:42.970451 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.970599 (+ 148us) service_pool.cc:224] Handling call
0924 19:57:44.724799 (+1754200us) ts_tablet_manager.cc:1151] Deleting tablet 44199d2655f14dcf97afe2180789aae4
0924 19:57:44.724845 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.778130 (+ 53285us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.851114 (+ 72984us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:44.876208 (+ 25094us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1750156}
I20260924 19:57:44.877669 18231 tablet_replica.cc:333] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:44.878492 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 8523125e35554830aff04b17677d5ceb (DEFAULT_TABLE table=default.Table_5e789131162643caa4c014a55244066f [id=52ff52bfd3904ab0b4826673127f70ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.879499 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8523125e35554830aff04b17677d5ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.879779 17950 tablet_service.cc:1511] Processing CreateTablet for tablet 8523125e35554830aff04b17677d5ceb (DEFAULT_TABLE table=default.Table_5e789131162643caa4c014a55244066f [id=52ff52bfd3904ab0b4826673127f70ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.879881 17798 tablet_service.cc:1511] Processing CreateTablet for tablet 8523125e35554830aff04b17677d5ceb (DEFAULT_TABLE table=default.Table_5e789131162643caa4c014a55244066f [id=52ff52bfd3904ab0b4826673127f70ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.880740 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8523125e35554830aff04b17677d5ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.880728 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8523125e35554830aff04b17677d5ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.883579 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 44199d2655f14dcf97afe2180789aae4 (table default.Table_1b2f1b739e234953861c274a098c2e1c [id=f02f0018bf7346c6a0ba91d52909be98]) successfully deleted
I20260924 19:57:44.884456 18231 raft_consensus.cc:2243] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:44.884582 18220 tablet_bootstrap.cc:654] T 92c0daa7d67e4aa390e334c669adda76 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.885021 18231 raft_consensus.cc:2272] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:44.890084 17613 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]). Expected version 1 got 0
I20260924 19:57:44.892297 18231 ts_tablet_manager.cc:1916] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:44.907423 19080 raft_consensus.cc:493] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:44.907968 19080 raft_consensus.cc:515] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.910311 18224 tablet_bootstrap.cc:492] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:44.910384 19080 leader_election.cc:290] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:44.910843 18224 ts_tablet_manager.cc:1403] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.117s user 0.017s sys 0.004s
I20260924 19:57:44.911919 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df722b0bbfd04ce2943f0e5b8058c50f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:44.913174 18224 raft_consensus.cc:359] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:44.913830 18050 leader_election.cc:343] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:44.918010 17967 raft_consensus.cc:3060] T affce4de8e0b412aa3b4bf19f9be7b6e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:44.915616 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df722b0bbfd04ce2943f0e5b8058c50f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:44.923269 18224 raft_consensus.cc:385] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.923509 17816 raft_consensus.cc:2468] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:44.923637 18224 raft_consensus.cc:740] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.924801 18051 leader_election.cc:304] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:44.925946 18224 consensus_queue.cc:260] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.926420 18090 tablet_service.cc:1511] Processing CreateTablet for tablet fbd5915ee98a4eda94479e9330cf8d04 (DEFAULT_TABLE table=default.Table_83714385573a44b2af65ac658fbe2cfa [id=e5b211f0af7143709eec620ecf04f2b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.926877 19080 raft_consensus.cc:2804] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:44.927234 19080 raft_consensus.cc:493] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:44.927549 19080 raft_consensus.cc:3060] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:44.927824 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbd5915ee98a4eda94479e9330cf8d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.927947 17614 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:44.929194 17614 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]). Expected version 2 got 0
I20260924 19:57:44.929728 17614 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:44.937315 18224 ts_tablet_manager.cc:1434] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260924 19:57:44.938810 18224 tablet_bootstrap.cc:492] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:44.940987 17798 tablet_service.cc:1511] Processing CreateTablet for tablet fbd5915ee98a4eda94479e9330cf8d04 (DEFAULT_TABLE table=default.Table_83714385573a44b2af65ac658fbe2cfa [id=e5b211f0af7143709eec620ecf04f2b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.942039 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbd5915ee98a4eda94479e9330cf8d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.942072 17940 tablet_service.cc:1511] Processing CreateTablet for tablet 57102b1727e94a90981509c82c082441 (DEFAULT_TABLE table=default.Table_2a8be69ab9604081a07df5197e7fab9d [id=8d382cdb38764b11ba323546fff603bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.943195 17940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57102b1727e94a90981509c82c082441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.945675 18231 ts_tablet_manager.cc:1929] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:44.949177 17614 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_3b871d019f0d40c4b916ef34227f5730 [id=fa469a44fb3741d39036337129cc36a1]). Expected version 1 got 0
I20260924 19:57:44.951465 18223 tablet_bootstrap.cc:654] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.951921 18231 log.cc:1199] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/3192f66a492442c1bf4137cfe7e13bc9
I20260924 19:57:44.954576 18231 ts_tablet_manager.cc:1950] T 3192f66a492442c1bf4137cfe7e13bc9 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:44.959398 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 3192f66a492442c1bf4137cfe7e13bc9 (table default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b [id=7babe5418e104e76871835ca3e30f95c]) successfully deleted
I20260924 19:57:44.961560 18095 tablet_service.cc:1511] Processing CreateTablet for tablet 57102b1727e94a90981509c82c082441 (DEFAULT_TABLE table=default.Table_2a8be69ab9604081a07df5197e7fab9d [id=8d382cdb38764b11ba323546fff603bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.956815 18224 tablet_bootstrap.cc:654] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:44.962538 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57102b1727e94a90981509c82c082441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.963194 19080 raft_consensus.cc:515] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:44.965801 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df722b0bbfd04ce2943f0e5b8058c50f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
W20260924 19:57:44.967133 18050 leader_election.cc:343] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:44.971967 17945 tablet_service.cc:1511] Processing CreateTablet for tablet fbd5915ee98a4eda94479e9330cf8d04 (DEFAULT_TABLE table=default.Table_83714385573a44b2af65ac658fbe2cfa [id=e5b211f0af7143709eec620ecf04f2b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.973335 18224 tablet_bootstrap.cc:492] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:44.973726 18224 ts_tablet_manager.cc:1403] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.000s
I20260924 19:57:44.975950 18224 raft_consensus.cc:359] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:44.977541 17799 tablet_service.cc:1511] Processing CreateTablet for tablet 57102b1727e94a90981509c82c082441 (DEFAULT_TABLE table=default.Table_2a8be69ab9604081a07df5197e7fab9d [id=8d382cdb38764b11ba323546fff603bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.978523 17799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57102b1727e94a90981509c82c082441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.978952 17947 tablet_service.cc:1511] Processing CreateTablet for tablet 2de3877b94b44c94b6d0fdad93f8aa79 (DEFAULT_TABLE table=default.Table_65d20bb187ea4069b2e589e7643028be [id=00e4a8cdcb374ae6a4446b3b7404d7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.981361 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de3877b94b44c94b6d0fdad93f8aa79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.981626 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 2de3877b94b44c94b6d0fdad93f8aa79 (DEFAULT_TABLE table=default.Table_65d20bb187ea4069b2e589e7643028be [id=00e4a8cdcb374ae6a4446b3b7404d7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.984325 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 2de3877b94b44c94b6d0fdad93f8aa79 (DEFAULT_TABLE table=default.Table_65d20bb187ea4069b2e589e7643028be [id=00e4a8cdcb374ae6a4446b3b7404d7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:44.985376 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de3877b94b44c94b6d0fdad93f8aa79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.988026 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbd5915ee98a4eda94479e9330cf8d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:44.993265 17611 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:44.994464 17611 catalog_manager.cc:5838] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
W20260924 19:57:44.995249 18050 consensus_peers.cc:597] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:44.997550 18224 raft_consensus.cc:385] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:44.997849 18224 raft_consensus.cc:740] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:44.999140 18223 tablet_bootstrap.cc:492] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:44.999577 18223 ts_tablet_manager.cc:1403] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.162s user 0.010s sys 0.000s
I20260924 19:57:45.003427 18224 consensus_queue.cc:260] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:45.010269 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2de3877b94b44c94b6d0fdad93f8aa79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.013950 18223 raft_consensus.cc:359] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.015095 18224 ts_tablet_manager.cc:1434] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20260924 19:57:45.018579 19051 tablet_replica.cc:333] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:45.014612 18223 raft_consensus.cc:385] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:45.019127 18223 raft_consensus.cc:740] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:45.019622 19051 raft_consensus.cc:2243] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:45.020247 19051 raft_consensus.cc:2272] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:45.019902 18223 consensus_queue.cc:260] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.022253 18223 ts_tablet_manager.cc:1434] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.022s user 0.004s sys 0.001s
I20260924 19:57:45.024353 18223 tablet_bootstrap.cc:492] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:45.031351 17798 tablet_service.cc:1511] Processing CreateTablet for tablet 70fea649583648ca86b4b3673ab0305f (DEFAULT_TABLE table=default.Table_66edcc97f7a64dfa85bf24d26e49c5de [id=31d1abb2c65744d78c5e61dd4ae81b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.031916 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 70fea649583648ca86b4b3673ab0305f (DEFAULT_TABLE table=default.Table_66edcc97f7a64dfa85bf24d26e49c5de [id=31d1abb2c65744d78c5e61dd4ae81b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.032981 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70fea649583648ca86b4b3673ab0305f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.033422 18223 tablet_bootstrap.cc:654] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:45.039534 17611 catalog_manager.cc:5838] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:45.042130 17611 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:45.043824 18223 tablet_bootstrap.cc:492] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:45.044210 18223 ts_tablet_manager.cc:1403] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.013s
I20260924 19:57:45.046489 18223 raft_consensus.cc:359] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.047084 18223 raft_consensus.cc:385] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:45.047348 18965 raft_consensus.cc:493] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.047873 18965 raft_consensus.cc:515] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.050150 18084 tablet_service.cc:1511] Processing CreateTablet for tablet e59e737a68dd443f9898113b2b2a401e (DEFAULT_TABLE table=default.Table_1dad27c8fd4445fb89e32722eef0abe2 [id=4cd61b434f884eed908abf700d02cdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.051100 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59e737a68dd443f9898113b2b2a401e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.055308 17802 tablet_service.cc:1511] Processing CreateTablet for tablet e59e737a68dd443f9898113b2b2a401e (DEFAULT_TABLE table=default.Table_1dad27c8fd4445fb89e32722eef0abe2 [id=4cd61b434f884eed908abf700d02cdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.063226 17611 catalog_manager.cc:5838] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:45.065415 18965 leader_election.cc:290] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:45.065251 17611 catalog_manager.cc:5838] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:45.066488 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f43f2a255c644ca6a68a8d867798fc1a" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:45.047365 18223 raft_consensus.cc:740] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:45.067354 18223 consensus_queue.cc:260] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.068217 17816 raft_consensus.cc:2468] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:45.074967 17795 tablet_service.cc:1511] Processing CreateTablet for tablet 18bb8f3d43df4aae8614b6dba9ffc668 (DEFAULT_TABLE table=default.Table_2255ca45bfcf4f819376770df3186e24 [id=fb5bd2d87d78475d803e753473de0c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.075668 17796 tablet_service.cc:1511] Processing CreateTablet for tablet 144355b18c2f413baa28dd632623db52 (DEFAULT_TABLE table=default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b [id=7babe5418e104e76871835ca3e30f95c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.069186 18051 leader_election.cc:304] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:45.072782 18095 tablet_service.cc:1511] Processing CreateTablet for tablet 18bb8f3d43df4aae8614b6dba9ffc668 (DEFAULT_TABLE table=default.Table_2255ca45bfcf4f819376770df3186e24 [id=fb5bd2d87d78475d803e753473de0c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.077848 18091 tablet_service.cc:1511] Processing CreateTablet for tablet 144355b18c2f413baa28dd632623db52 (DEFAULT_TABLE table=default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b [id=7babe5418e104e76871835ca3e30f95c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.080525 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18bb8f3d43df4aae8614b6dba9ffc668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.081104 18091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144355b18c2f413baa28dd632623db52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.081455 18951 raft_consensus.cc:2804] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:45.081799 18951 raft_consensus.cc:493] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:45.082098 18951 raft_consensus.cc:3060] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:45.087730 17611 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]). Expected version 1 got 0
W20260924 19:57:45.089141 17607 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e6d926ff7c24901a15b0c7b2239fbad consensus state has no leader
I20260924 19:57:45.089636 17607 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 246) took 1456 ms. Trace:
I20260924 19:57:45.088513 17610 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3] alter complete (version 2)
I20260924 19:57:45.091408 17609 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 9c861e93ad9f40ba9ad010021e43bc2b
I20260924 19:57:45.089926 17607 rpcz_store.cc:276] 0924 19:57:43.633081 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:44.101408 (+468327us) service_pool.cc:224] Handling call
0924 19:57:44.102278 (+ 870us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:44.102891 (+ 613us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:44.114160 (+ 11269us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:44.114397 (+ 237us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:44.311746 (+197349us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:45.089578 (+777832us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:44.116575 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:44.117049 (+ 474us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:44.117076 (+ 27us) write_op.cc:435] Acquired schema lock
0924 19:57:44.117092 (+ 16us) tablet.cc:662] Decoding operations
0924 19:57:44.117374 (+ 282us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:44.117410 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:44.117435 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:44.117609 (+ 174us) tablet.cc:701] Row locks acquired
0924 19:57:44.117626 (+ 17us) write_op.cc:260] PREPARE: finished
0924 19:57:44.117738 (+ 112us) write_op.cc:270] Start()
0924 19:57:44.117828 (+ 90us) write_op.cc:276] Timestamp: P: 1790279864117667 usec, L: 0
0924 19:57:44.117846 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:44.118409 (+ 563us) log.cc:844] Serialized 405 byte log entry
0924 19:57:44.307306 (+188897us) op_driver.cc:464] REPLICATION: finished
0924 19:57:44.309555 (+ 2249us) write_op.cc:301] APPLY: starting
0924 19:57:44.309602 (+ 47us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:44.309685 (+ 83us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:44.309778 (+ 93us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:44.309905 (+ 127us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:44.310029 (+ 124us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:44.310066 (+ 37us) write_op.cc:312] APPLY: finished
0924 19:57:44.310215 (+ 149us) log.cc:844] Serialized 23 byte log entry
0924 19:57:44.310322 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:44.310378 (+ 56us) write_op.cc:454] Released schema lock
0924 19:57:44.310404 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:44.310616 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:44.310638 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2128,"apply.run_cpu_time_us":1030,"apply.run_wall_time_us":1030,"num_ops":1,"prepare.queue_time_us":118,"prepare.run_cpu_time_us":2050,"prepare.run_wall_time_us":2049,"replication_time_us":189382}]]}
I20260924 19:57:45.092942 17609 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 237) took 1920 ms. Trace:
I20260924 19:57:45.101980 18780 raft_consensus.cc:493] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.102124 17797 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c861e93ad9f40ba9ad010021e43bc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:43 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.101652 17608 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_7b49c63ad79c4132afbbbb9e1f2d4ca0 [id=8edea4510fd84d40819b48b401ccfe64] alter complete (version 3)
I20260924 19:57:45.102464 18780 raft_consensus.cc:515] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:45.104439 18780 leader_election.cc:290] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:45.093246 17609 rpcz_store.cc:276] 0924 19:57:43.173020 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:43.173145 (+ 125us) service_pool.cc:224] Handling call
0924 19:57:43.182939 (+ 9794us) spinlock_profiling.cc:243] Waited 9.1 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552207476 00007fa54fdb7b34 00007fa54fdb7ac2 00007fa54fdb79f1 00007fa54fdb796c 00007fa54fdb6c22 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:43.183928 (+ 989us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:43.184411 (+ 483us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:43.184577 (+ 166us) catalog_manager.cc:2863] Locking tablets
0924 19:57:43.185123 (+ 546us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:43.380833 (+195710us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:43.381019 (+ 186us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:45.091315 (+1710296us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:45.092885 (+ 1570us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:43.211658 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:43.212134 (+ 476us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:43.212165 (+ 31us) write_op.cc:435] Acquired schema lock
0924 19:57:43.212178 (+ 13us) tablet.cc:662] Decoding operations
0924 19:57:43.232767 (+ 20589us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:43.232809 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:43.232837 (+ 28us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:43.233159 (+ 322us) tablet.cc:701] Row locks acquired
0924 19:57:43.233176 (+ 17us) write_op.cc:260] PREPARE: finished
0924 19:57:43.233311 (+ 135us) write_op.cc:270] Start()
0924 19:57:43.328392 (+ 95081us) write_op.cc:276] Timestamp: P: 1790279863233227 usec, L: 0
0924 19:57:43.328413 (+ 21us) op_driver.cc:348] REPLICATION: starting
0924 19:57:43.329127 (+ 714us) log.cc:844] Serialized 793 byte log entry
0924 19:57:43.375461 (+ 46334us) op_driver.cc:464] REPLICATION: finished
0924 19:57:43.377851 (+ 2390us) write_op.cc:301] APPLY: starting
0924 19:57:43.377908 (+ 57us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:43.378011 (+ 103us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:43.378099 (+ 88us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:43.378386 (+ 287us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:43.378520 (+ 134us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:43.378559 (+ 39us) write_op.cc:312] APPLY: finished
0924 19:57:43.378733 (+ 174us) log.cc:844] Serialized 29 byte log entry
0924 19:57:43.378886 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:43.378960 (+ 74us) write_op.cc:454] Released schema lock
0924 19:57:43.378992 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:43.379239 (+ 247us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:43.379266 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":20938496,"child_traces":[["op",{"apply.queue_time_us":2246,"apply.run_cpu_time_us":1359,"apply.run_wall_time_us":1359,"num_ops":2,"prepare.queue_time_us":24711,"prepare.run_cpu_time_us":2803,"prepare.run_wall_time_us":120150,"raft.queue_time_us":97,"raft.run_cpu_time_us":858,"raft.run_wall_time_us":857,"replication_time_us":46933}]]}
I20260924 19:57:44.989564 17948 tablet_service.cc:1511] Processing CreateTablet for tablet 70fea649583648ca86b4b3673ab0305f (DEFAULT_TABLE table=default.Table_66edcc97f7a64dfa85bf24d26e49c5de [id=31d1abb2c65744d78c5e61dd4ae81b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.018080 17966 raft_consensus.cc:3060] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:45.127144 18951 raft_consensus.cc:515] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.130045 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f43f2a255c644ca6a68a8d867798fc1a" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:45.130265 17613 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_a7d63063e7ee4493abe8c76da6d4f99d [id=be19c62729a34ae1a15966662303a869] alter complete (version 2)
I20260924 19:57:45.130622 17817 raft_consensus.cc:3060] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:45.135298 17615 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 225) took 2520 ms. Trace:
I20260924 19:57:45.135607 17615 rpcz_store.cc:276] 0924 19:57:42.615339 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.615475 (+ 136us) service_pool.cc:224] Handling call
0924 19:57:42.636031 (+ 20556us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa5522077e6 00007fa54fdbc634 00007fa54fdbc5c2 00007fa54fdbc4f1 00007fa54fdbc46c 00007fa54fdbb722 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:42.636762 (+ 731us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:42.637375 (+ 613us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:42.637970 (+ 595us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:42.638132 (+ 162us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:42.712923 (+ 74791us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:45.135237 (+2422314us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:42.641662 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:42.642087 (+ 425us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:42.642112 (+ 25us) write_op.cc:435] Acquired schema lock
0924 19:57:42.642125 (+ 13us) tablet.cc:662] Decoding operations
0924 19:57:42.642403 (+ 278us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:42.642441 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:42.642465 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:42.642616 (+ 151us) tablet.cc:701] Row locks acquired
0924 19:57:42.642631 (+ 15us) write_op.cc:260] PREPARE: finished
0924 19:57:42.642752 (+ 121us) write_op.cc:270] Start()
0924 19:57:42.642833 (+ 81us) write_op.cc:276] Timestamp: P: 1790279862642618 usec, L: 0
0924 19:57:42.642851 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:42.644773 (+ 1922us) log.cc:844] Serialized 405 byte log entry
0924 19:57:42.680367 (+ 35594us) op_driver.cc:464] REPLICATION: finished
0924 19:57:42.697039 (+ 16672us) write_op.cc:301] APPLY: starting
0924 19:57:42.697092 (+ 53us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:42.697172 (+ 80us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:42.697254 (+ 82us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:42.697424 (+ 170us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:42.697551 (+ 127us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:42.697596 (+ 45us) write_op.cc:312] APPLY: finished
0924 19:57:42.697747 (+ 151us) log.cc:844] Serialized 23 byte log entry
0924 19:57:42.712670 (+ 14923us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.712729 (+ 59us) write_op.cc:454] Released schema lock
0924 19:57:42.712757 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:42.713229 (+ 472us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.713253 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":198,"spinlock_wait_cycles":40951168,"child_traces":[["op",{"apply.queue_time_us":16547,"apply.run_cpu_time_us":1129,"apply.run_wall_time_us":16143,"num_ops":1,"prepare.queue_time_us":2450,"prepare.run_cpu_time_us":1917,"prepare.run_wall_time_us":3305,"replication_time_us":37446,"thread_start_us":21503,"threads_started":1}]]}
I20260924 19:57:45.139654 17817 raft_consensus.cc:2468] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:45.140614 17943 tablet_service.cc:1558] Processing DeleteTablet for tablet 83ee43b6dc064dc7b320691d52e57ea7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c1afbfc7de7480ab4be40680a3e3ada) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:45.144292 17948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70fea649583648ca86b4b3673ab0305f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.144826 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_5cc282e71cd74150a849aff1620d54f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:45.152086 17943 tablet_service.cc:1558] Processing DeleteTablet for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:45.152593 17941 tablet_service.cc:1558] Processing DeleteTablet for tablet 910d61b8c3574302b5a328187226571e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:40 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:45.152774 17942 tablet_service.cc:1558] Processing DeleteTablet for tablet dd2f4eb8bf254c4e85b6a9072271766b with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1ac7e1b192465696640c02898bc39f) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:45.153592 17942 tablet_service.cc:1558] Processing DeleteTablet for tablet f90a677e94034c73ab5452554862c1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:40 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:45.158842 18051 leader_election.cc:304] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:45.161052 18220 tablet_bootstrap.cc:492] T 92c0daa7d67e4aa390e334c669adda76 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:45.161484 18220 ts_tablet_manager.cc:1403] T 92c0daa7d67e4aa390e334c669adda76 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.456s user 0.008s sys 0.000s
I20260924 19:57:45.163753 17942 tablet_service.cc:1558] Processing DeleteTablet for tablet a765abd702644f079bddffb3fc4238ef with delete_type TABLET_DATA_DELETED (replaced by tablet f169c47d711d4566b2004cdfd0de725d) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:45.166474 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cc282e71cd74150a849aff1620d54f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:45.170312 18780 raft_consensus.cc:493] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.170801 18780 raft_consensus.cc:515] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:45.172036 19080 leader_election.cc:290] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:45.172803 18780 leader_election.cc:290] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:45.172348 18965 raft_consensus.cc:2804] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:45.173545 18965 raft_consensus.cc:697] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:45.176242 17942 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a524bad1c4040e58ad84fcf5a4fc47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:37 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:45.178351 18224 tablet_bootstrap.cc:492] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:45.180160 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57008266b80345e0a3f8e60bd7e2d62b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:45.181355 17753 leader_election.cc:343] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.183383 17798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70fea649583648ca86b4b3673ab0305f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.187752 17802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59e737a68dd443f9898113b2b2a401e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.191212 17613 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140] alter complete (version 2)
I20260924 19:57:45.191612 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18bb8f3d43df4aae8614b6dba9ffc668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.192077 17796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144355b18c2f413baa28dd632623db52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.192466 18231 tablet_replica.cc:333] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:45.193210 18231 raft_consensus.cc:2243] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:45.193666 18231 raft_consensus.cc:2272] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:45.200773 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet dd2f4eb8bf254c4e85b6a9072271766b with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1ac7e1b192465696640c02898bc39f) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.202491 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet f90a677e94034c73ab5452554862c1c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:40 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.203204 17797 tablet_service.cc:1558] Processing DeleteTablet for tablet 83ee43b6dc064dc7b320691d52e57ea7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7c1afbfc7de7480ab4be40680a3e3ada) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.045064 17950 tablet_service.cc:1511] Processing CreateTablet for tablet e59e737a68dd443f9898113b2b2a401e (DEFAULT_TABLE table=default.Table_1dad27c8fd4445fb89e32722eef0abe2 [id=4cd61b434f884eed908abf700d02cdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.056838 18910 ts_tablet_manager.cc:1929] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:45.204103 18910 log.cc:1199] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/d970cc92a1134a98a613d198a319cb01
I20260924 19:57:45.204689 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59e737a68dd443f9898113b2b2a401e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.204839 18910 ts_tablet_manager.cc:1950] T d970cc92a1134a98a613d198a319cb01 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:45.205367 18780 raft_consensus.cc:493] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.205763 18780 raft_consensus.cc:515] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.206741 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 199) took 2987 ms. Trace:
I20260924 19:57:45.057803 17946 tablet_service.cc:1511] Processing CreateTablet for tablet 18bb8f3d43df4aae8614b6dba9ffc668 (DEFAULT_TABLE table=default.Table_2255ca45bfcf4f819376770df3186e24 [id=fb5bd2d87d78475d803e753473de0c7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.068001 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f43f2a255c644ca6a68a8d867798fc1a" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:45.077594 17944 tablet_service.cc:1511] Processing CreateTablet for tablet 144355b18c2f413baa28dd632623db52 (DEFAULT_TABLE table=default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b [id=7babe5418e104e76871835ca3e30f95c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.101379 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c861e93ad9f40ba9ad010021e43bc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:43 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:45.106943 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d95d6e48173436c8c59130464413ea1" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:45.209193 18780 leader_election.cc:290] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:45.209077 17942 tablet_service.cc:1511] Processing CreateTablet for tablet bea1ebdf647e4ef9a131a0398672b8ab (DEFAULT_TABLE table=default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.210167 18231 ts_tablet_manager.cc:1916] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:45.211099 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 07f86f24d9d14d05898e1c0f69428850 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d95d6e48173436c8c59130464413ea1) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:45.211704 17753 leader_election.cc:343] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.211835 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca37ddd22f5d4bcab2862553408447ce" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:45.212141 17614 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 42) took 1634 ms (client timeout 1000 ms). Trace:
W20260924 19:57:45.212462 17614 rpcz_store.cc:269] 0924 19:57:43.577672 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:43.667405 (+ 89733us) service_pool.cc:224] Handling call
0924 19:57:45.212057 (+1544652us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":27}
W20260924 19:57:45.213035 17753 leader_election.cc:343] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.207028 18910 rpcz_store.cc:276] 0924 19:57:42.218906 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.219066 (+ 160us) service_pool.cc:224] Handling call
0924 19:57:44.870926 (+2651860us) ts_tablet_manager.cc:1151] Deleting tablet d970cc92a1134a98a613d198a319cb01
0924 19:57:44.870965 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:44.971912 (+100947us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:45.056702 (+ 84790us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:45.206682 (+149980us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2651195}
I20260924 19:57:45.217809 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 204) took 2611 ms. Trace:
I20260924 19:57:45.218103 18910 rpcz_store.cc:276] 0924 19:57:42.606064 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.606193 (+ 129us) service_pool.cc:224] Handling call
0924 19:57:45.217486 (+2611293us) ts_tablet_manager.cc:1151] Deleting tablet ae7a0660795e4e4b87d0959ca84d5d1b
0924 19:57:45.217535 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:45.217761 (+ 226us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2607161}
I20260924 19:57:45.221446 17797 tablet_service.cc:1511] Processing CreateTablet for tablet bea1ebdf647e4ef9a131a0398672b8ab (DEFAULT_TABLE table=default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.222597 17797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea1ebdf647e4ef9a131a0398672b8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.223415 17946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18bb8f3d43df4aae8614b6dba9ffc668. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.224634 18220 raft_consensus.cc:359] T 92c0daa7d67e4aa390e334c669adda76 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.225262 18220 raft_consensus.cc:385] T 92c0daa7d67e4aa390e334c669adda76 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:45.225512 18220 raft_consensus.cc:740] T 92c0daa7d67e4aa390e334c669adda76 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:45.226197 18220 consensus_queue.cc:260] T 92c0daa7d67e4aa390e334c669adda76 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.227466 17942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea1ebdf647e4ef9a131a0398672b8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.228037 18220 ts_tablet_manager.cc:1434] T 92c0daa7d67e4aa390e334c669adda76 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.066s user 0.006s sys 0.000s
I20260924 19:57:45.192310 19051 ts_tablet_manager.cc:1916] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:45.228833 18965 consensus_queue.cc:237] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.231613 17944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 144355b18c2f413baa28dd632623db52. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:45.216820 18050 leader_election.cc:343] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.234925 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d95d6e48173436c8c59130464413ea1" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:45.217669 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet d970cc92a1134a98a613d198a319cb01 (table default.Table_a50429edeaf645668bc84244412bb359 [id=1853f611de514a55afd7a68d742503b8]) successfully deleted
I20260924 19:57:45.234754 19080 consensus_queue.cc:1048] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:45.224303 17606 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 41) took 2799 ms (client timeout 1000 ms). Trace:
W20260924 19:57:45.236410 17606 rpcz_store.cc:269] 0924 19:57:42.425324 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.501186 (+ 75862us) service_pool.cc:224] Handling call
0924 19:57:42.653499 (+152313us) spinlock_profiling.cc:243] Waited 35.5 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:45.224235 (+2570736us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":452,"reported_tablets":26,"spinlock_wait_cycles":81585152}
I20260924 19:57:45.236712 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57008266b80345e0a3f8e60bd7e2d62b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:45.238147 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_a36de9161f8f4ea08243a7222c099822"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:45.239334 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca37ddd22f5d4bcab2862553408447ce" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:45.239743 18090 tablet_service.cc:1558] Processing DeleteTablet for tablet 910d61b8c3574302b5a328187226571e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:40 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:45.240725 18090 tablet_service.cc:1558] Processing DeleteTablet for tablet 07f86f24d9d14d05898e1c0f69428850 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d95d6e48173436c8c59130464413ea1) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:45.218242 18223 ts_tablet_manager.cc:1434] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.174s user 0.006s sys 0.000s
I20260924 19:57:45.241815 18223 tablet_bootstrap.cc:492] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:45.228576 17610 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 31) took 3503 ms (client timeout 1000 ms). Trace:
I20260924 19:57:45.221647 18951 leader_election.cc:290] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:45.244872 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 86107.4us
I20260924 19:57:45.248850 18224 tablet_bootstrap.cc:654] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:45.239102 17614 master_service.cc:960] ReplaceTablet: received request to replace tablet 201ea0020cb3430eac85e80b8c06fb07 from {username='slave'} at 127.0.0.1:51280
W20260924 19:57:45.227365 17612 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 32) took 2096 ms (client timeout 1000 ms). Trace:
W20260924 19:57:45.255746 17612 rpcz_store.cc:269] 0924 19:57:43.130556 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:43.130698 (+ 142us) service_pool.cc:224] Handling call
0924 19:57:45.227298 (+2096600us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:44.442955 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:44.443443 (+ 488us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:44.443471 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:57:44.443486 (+ 15us) tablet.cc:662] Decoding operations
0924 19:57:44.443855 (+ 369us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:44.443906 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:44.443932 (+ 26us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:44.444232 (+ 300us) tablet.cc:701] Row locks acquired
0924 19:57:44.444245 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:57:44.444369 (+ 124us) write_op.cc:270] Start()
0924 19:57:44.444471 (+ 102us) write_op.cc:276] Timestamp: P: 1790279864444331 usec, L: 0
0924 19:57:44.444489 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:44.445035 (+ 546us) log.cc:844] Serialized 907 byte log entry
0924 19:57:44.477091 (+ 32056us) op_driver.cc:464] REPLICATION: finished
0924 19:57:44.477495 (+ 404us) write_op.cc:301] APPLY: starting
0924 19:57:44.477563 (+ 68us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:44.477682 (+ 119us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:44.477785 (+ 103us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:44.478109 (+ 324us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:44.478242 (+ 133us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:44.478284 (+ 42us) write_op.cc:312] APPLY: finished
0924 19:57:44.478466 (+ 182us) log.cc:844] Serialized 29 byte log entry
0924 19:57:44.478606 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:44.478677 (+ 71us) write_op.cc:454] Released schema lock
0924 19:57:44.478707 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:44.478933 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:44.478960 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":135527,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":242,"apply.run_cpu_time_us":1412,"apply.run_wall_time_us":1411,"num_ops":2,"prepare.queue_time_us":112,"prepare.run_cpu_time_us":2379,"prepare.run_wall_time_us":2381,"raft.queue_time_us":1247,"raft.run_cpu_time_us":773,"raft.run_wall_time_us":772,"replication_time_us":32499}]]}
I20260924 19:57:45.259478 17612 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_8ade2ea47f344fd18a687f7d5e0ea568"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:45.265820 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 910d61b8c3574302b5a328187226571e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:40 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.266484 17799 tablet_service.cc:1558] Processing DeleteTablet for tablet ba46ac86b242407abd3e8dbf09f12efa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.267082 17801 tablet_service.cc:1558] Processing DeleteTablet for tablet 07f86f24d9d14d05898e1c0f69428850 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d95d6e48173436c8c59130464413ea1) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.258941 18090 tablet_service.cc:1558] Processing DeleteTablet for tablet a765abd702644f079bddffb3fc4238ef with delete_type TABLET_DATA_DELETED (replaced by tablet f169c47d711d4566b2004cdfd0de725d) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:45.267741 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 44199d2655f14dcf97afe2180789aae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.269842 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df722b0bbfd04ce2943f0e5b8058c50f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:45.271226 17816 raft_consensus.cc:3060] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:45.271656 19135 raft_consensus.cc:493] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:45.271807 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet ae7a0660795e4e4b87d0959ca84d5d1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae7a0660795e4e4b87d0959ca84d5d1b already in progress: creating tablet
I20260924 19:57:45.272238 19135 raft_consensus.cc:515] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.275084 19135 leader_election.cc:290] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:45.276239 18082 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c861e93ad9f40ba9ad010021e43bc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:43 UTC) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:45.244076 17610 rpcz_store.cc:269] 0924 19:57:41.725638 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.070458 (+344820us) service_pool.cc:224] Handling call
0924 19:57:45.228515 (+3158057us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:42.503361 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:42.503816 (+ 455us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:42.503841 (+ 25us) write_op.cc:435] Acquired schema lock
0924 19:57:42.503853 (+ 12us) tablet.cc:662] Decoding operations
0924 19:57:42.504136 (+ 283us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:42.504177 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:42.504202 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:42.504364 (+ 162us) tablet.cc:701] Row locks acquired
0924 19:57:42.504378 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:42.504497 (+ 119us) write_op.cc:270] Start()
0924 19:57:42.504586 (+ 89us) write_op.cc:276] Timestamp: P: 1790279862504385 usec, L: 0
0924 19:57:42.504604 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:42.505139 (+ 535us) log.cc:844] Serialized 530 byte log entry
0924 19:57:42.625536 (+120397us) op_driver.cc:464] REPLICATION: finished
0924 19:57:42.629476 (+ 3940us) write_op.cc:301] APPLY: starting
0924 19:57:42.629540 (+ 64us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:42.629634 (+ 94us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:42.629725 (+ 91us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:42.629910 (+ 185us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:42.630030 (+ 120us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:42.630068 (+ 38us) write_op.cc:312] APPLY: finished
0924 19:57:42.630230 (+ 162us) log.cc:844] Serialized 23 byte log entry
0924 19:57:42.630758 (+ 528us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.630815 (+ 57us) write_op.cc:454] Released schema lock
0924 19:57:42.630846 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:42.645827 (+ 14981us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.645865 (+ 38us) write_op.cc:454] Released schema lock
Related trace 'op':
0924 19:57:42.639503 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:42.639994 (+ 491us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:42.640025 (+ 31us) write_op.cc:435] Acquired schema lock
0924 19:57:42.640039 (+ 14us) tablet.cc:662] Decoding operations
0924 19:57:42.640320 (+ 281us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:42.640363 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:42.640398 (+ 35us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:42.640560 (+ 162us) tablet.cc:701] Row locks acquired
0924 19:57:42.640573 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:57:42.640696 (+ 123us) write_op.cc:270] Start()
0924 19:57:42.640789 (+ 93us) write_op.cc:276] Timestamp: P: 1790279862640572 usec, L: 0
0924 19:57:42.640806 (+ 17us) op_driver.cc:348] REPLICATION: starting
0924 19:57:42.641316 (+ 510us) log.cc:844] Serialized 356 byte log entry
0924 19:57:42.680143 (+ 38827us) op_driver.cc:464] REPLICATION: finished
0924 19:57:42.694663 (+ 14520us) write_op.cc:301] APPLY: starting
0924 19:57:42.694723 (+ 60us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:42.694812 (+ 89us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:42.694933 (+ 121us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:42.695106 (+ 173us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:42.695263 (+ 157us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:42.695309 (+ 46us) write_op.cc:312] APPLY: finished
0924 19:57:42.695493 (+ 184us) log.cc:844] Serialized 23 byte log entry
0924 19:57:42.711704 (+ 16211us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.711770 (+ 66us) write_op.cc:454] Released schema lock
0924 19:57:42.711802 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:42.712171 (+ 369us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:42.712197 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":54462,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":3788,"apply.run_cpu_time_us":1172,"apply.run_wall_time_us":1563,"num_ops":1,"prepare.queue_time_us":8926,"prepare.run_cpu_time_us":2003,"prepare.run_wall_time_us":2003,"replication_time_us":120857}],["op",{"apply.queue_time_us":14324,"apply.run_cpu_time_us":1343,"apply.run_wall_time_us":17489,"num_ops":1,"prepare.queue_time_us":3727,"prepare.run_cpu_time_us":2035,"prepare.run_wall_time_us":2035,"replication_time_us":39230}]]}
W20260924 19:57:45.265973 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a36de9161f8f4ea08243a7222c099822 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:45.289070 17940 tablet_service.cc:1511] Processing CreateTablet for tablet 97bbb17c170d42349d2c9cdb41349cb9 (DEFAULT_TABLE table=default.Table_b2824cbbb81c4186840b758c61eccbb2 [id=4e169b185baf4e7a81735a13b6d82fca]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:57:45.281824 17608 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 44) took 3018 ms (client timeout 1000 ms). Trace:
W20260924 19:57:45.297642 17612 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ade2ea47f344fd18a687f7d5e0ea568 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:57:45.308825 17613 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 45) took 1878 ms (client timeout 1000 ms). Trace:
I20260924 19:57:45.315598 17610 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_6ddb9314aa114575b5317b1d08ba138e" } modify_external_catalogs: true
I20260924 19:57:45.318846 17610 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_6ddb9314aa114575b5317b1d08ba138e" } modify_external_catalogs: true
I20260924 19:57:45.322948 17614 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 201ea0020cb3430eac85e80b8c06fb07
I20260924 19:57:45.324482 17615 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 248) took 1439 ms. Trace:
I20260924 19:57:45.325006 17614 catalog_manager.cc:6934] ReplaceTablet: tablet 201ea0020cb3430eac85e80b8c06fb07 deleted and replaced by tablet a00e3626ea154dc5a959f0fbb4e849dc
I20260924 19:57:45.328446 17614 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51280 (request call id 250) took 1136 ms. Trace:
W20260924 19:57:45.298022 17608 rpcz_store.cc:269] 0924 19:57:42.263222 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.263710 (+ 488us) service_pool.cc:224] Handling call
0924 19:57:45.281761 (+3018051us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:43.477265 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:43.477724 (+ 459us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:43.477749 (+ 25us) write_op.cc:435] Acquired schema lock
0924 19:57:43.477765 (+ 16us) tablet.cc:662] Decoding operations
0924 19:57:43.478035 (+ 270us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:43.478076 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:43.478100 (+ 24us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:43.478260 (+ 160us) tablet.cc:701] Row locks acquired
0924 19:57:43.478276 (+ 16us) write_op.cc:260] PREPARE: finished
0924 19:57:43.478403 (+ 127us) write_op.cc:270] Start()
0924 19:57:43.478498 (+ 95us) write_op.cc:276] Timestamp: P: 1790279863478344 usec, L: 0
0924 19:57:43.478514 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:43.479021 (+ 507us) log.cc:844] Serialized 356 byte log entry
0924 19:57:43.534998 (+ 55977us) op_driver.cc:464] REPLICATION: finished
0924 19:57:43.535294 (+ 296us) write_op.cc:301] APPLY: starting
0924 19:57:43.535355 (+ 61us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:43.535470 (+ 115us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:43.535581 (+ 111us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:43.535781 (+ 200us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:43.535924 (+ 143us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:43.535971 (+ 47us) write_op.cc:312] APPLY: finished
0924 19:57:43.536144 (+ 173us) log.cc:844] Serialized 23 byte log entry
0924 19:57:43.536307 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:43.536374 (+ 67us) write_op.cc:454] Released schema lock
0924 19:57:43.536416 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:43.536660 (+ 244us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:43.536700 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":29757,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":139,"apply.run_cpu_time_us":1328,"apply.run_wall_time_us":1327,"num_ops":1,"prepare.queue_time_us":149174,"prepare.run_cpu_time_us":2108,"prepare.run_wall_time_us":49872,"raft.queue_time_us":92,"raft.run_cpu_time_us":747,"raft.run_wall_time_us":747,"replication_time_us":56387}]]}
I20260924 19:57:45.324869 17615 rpcz_store.cc:276] 0924 19:57:43.884778 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.144466 (+1259688us) service_pool.cc:224] Handling call
0924 19:57:45.241523 (+ 97057us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:45.272760 (+ 31237us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:45.323726 (+ 50966us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:45.324115 (+ 389us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:45.324424 (+ 309us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:45.275692 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:45.276147 (+ 455us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:45.276175 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:57:45.276192 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:45.276491 (+ 299us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:45.276599 (+ 108us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:45.276627 (+ 28us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:45.276860 (+ 233us) tablet.cc:701] Row locks acquired
0924 19:57:45.276874 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:45.276996 (+ 122us) write_op.cc:270] Start()
0924 19:57:45.277100 (+ 104us) write_op.cc:276] Timestamp: P: 1790279865276921 usec, L: 0
0924 19:57:45.277115 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:57:45.277627 (+ 512us) log.cc:844] Serialized 433 byte log entry
0924 19:57:45.314630 (+ 37003us) op_driver.cc:464] REPLICATION: finished
0924 19:57:45.319368 (+ 4738us) write_op.cc:301] APPLY: starting
0924 19:57:45.319429 (+ 61us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:45.319543 (+ 114us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:45.319652 (+ 109us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:45.320126 (+ 474us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:45.320260 (+ 134us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:45.320306 (+ 46us) write_op.cc:312] APPLY: finished
0924 19:57:45.320489 (+ 183us) log.cc:844] Serialized 29 byte log entry
0924 19:57:45.320665 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.320751 (+ 86us) write_op.cc:454] Released schema lock
0924 19:57:45.320795 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:45.321029 (+ 234us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.321052 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4580,"apply.run_cpu_time_us":1625,"apply.run_wall_time_us":1625,"num_ops":2,"prepare.queue_time_us":1759,"prepare.run_cpu_time_us":2195,"prepare.run_wall_time_us":4032,"raft.queue_time_us":104,"raft.run_cpu_time_us":662,"raft.run_wall_time_us":669,"replication_time_us":37406}]]}
I20260924 19:57:45.334515 17615 master_service.cc:960] ReplaceTablet: received request to replace tablet c058a49bfd5544b58b76edfd3fadbd1a from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:45.336033 17941 tablet_service.cc:1558] Processing DeleteTablet for tablet 201ea0020cb3430eac85e80b8c06fb07 with delete_type TABLET_DATA_DELETED (replaced by tablet a00e3626ea154dc5a959f0fbb4e849dc) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:45.337760 17941 tablet_service.cc:1511] Processing CreateTablet for tablet 10fe8d2d06dd4f83bef8c52a82693344 (DEFAULT_TABLE table=default.Table_fcbe3b81791a4aaf923847b1366eb6f9 [id=25cb87a38f0f473d99bad6f06d4edcf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.345687 18951 consensus_queue.cc:1048] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:45.349274 18951 raft_consensus.cc:493] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.349706 18951 raft_consensus.cc:515] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.351761 18951 leader_election.cc:290] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:45.354734 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 205) took 2399 ms. Trace:
I20260924 19:57:45.354629 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f43f2a255c644ca6a68a8d867798fc1a" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:45.355064 18910 rpcz_store.cc:276] 0924 19:57:42.955570 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.955696 (+ 126us) service_pool.cc:224] Handling call
0924 19:57:45.219026 (+2263330us) ts_tablet_manager.cc:1151] Deleting tablet ba46ac86b242407abd3e8dbf09f12efa
0924 19:57:45.219065 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:45.354424 (+135359us) spinlock_profiling.cc:243] Waited 86.6 ms on lock 0x7b5800000f98. stack: 00007f271de83323 00007f271de83220 00007f271da27574 0000000000516964 00000000005168f9 00007f271dd398a1 00007f271de7db51 00007f271de7e2ee 00007f2726deff09 00007f27260029e1 00007f27260c9400 00007f27260c98c3 00007f27260fa960 00007f27260e0e6d 00007f27260e0e19 00007f27260e0da9
0924 19:57:45.354676 (+ 252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":199270656,"tablet-delete.queue_time_us":2262831}
W20260924 19:57:45.356681 18050 leader_election.cc:343] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.356897 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b99ff1920274831b58d8283bd592818" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:45.361541 18050 leader_election.cc:343] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.362049 17940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97bbb17c170d42349d2c9cdb41349cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.364249 17941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fe8d2d06dd4f83bef8c52a82693344. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:45.310977 17613 rpcz_store.cc:269] 0924 19:57:43.429921 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:43.430336 (+ 415us) service_pool.cc:224] Handling call
0924 19:57:43.452525 (+ 22189us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:45.308745 (+1856220us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:45.102920 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:45.103437 (+ 517us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:45.103467 (+ 30us) write_op.cc:435] Acquired schema lock
0924 19:57:45.103482 (+ 15us) tablet.cc:662] Decoding operations
0924 19:57:45.103772 (+ 290us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:45.103822 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:45.103847 (+ 25us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:45.104021 (+ 174us) tablet.cc:701] Row locks acquired
0924 19:57:45.104038 (+ 17us) write_op.cc:260] PREPARE: finished
0924 19:57:45.104165 (+ 127us) write_op.cc:270] Start()
0924 19:57:45.104259 (+ 94us) write_op.cc:276] Timestamp: P: 1790279865104098 usec, L: 0
0924 19:57:45.104279 (+ 20us) op_driver.cc:348] REPLICATION: starting
0924 19:57:45.104810 (+ 531us) log.cc:844] Serialized 356 byte log entry
0924 19:57:45.118429 (+ 13619us) op_driver.cc:464] REPLICATION: finished
0924 19:57:45.118754 (+ 325us) write_op.cc:301] APPLY: starting
0924 19:57:45.118813 (+ 59us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:45.118909 (+ 96us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:45.118998 (+ 89us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:45.119234 (+ 236us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:45.119424 (+ 190us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:45.119463 (+ 39us) write_op.cc:312] APPLY: finished
0924 19:57:45.119630 (+ 167us) log.cc:844] Serialized 23 byte log entry
0924 19:57:45.119764 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.119820 (+ 56us) write_op.cc:454] Released schema lock
0924 19:57:45.119849 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:45.120082 (+ 233us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.120105 (+ 23us) write_op.cc:454] Released schema lock
Related trace 'op':
0924 19:57:45.133033 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:45.133486 (+ 453us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:45.133513 (+ 27us) write_op.cc:435] Acquired schema lock
0924 19:57:45.133530 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:45.133801 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:45.134267 (+ 466us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:45.134296 (+ 29us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:45.153592 (+ 19296us) tablet.cc:701] Row locks acquired
0924 19:57:45.153607 (+ 15us) write_op.cc:260] PREPARE: finished
0924 19:57:45.153752 (+ 145us) write_op.cc:270] Start()
0924 19:57:45.153851 (+ 99us) write_op.cc:276] Timestamp: P: 1790279865153684 usec, L: 0
0924 19:57:45.153867 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:45.154444 (+ 577us) log.cc:844] Serialized 356 byte log entry
0924 19:57:45.164316 (+ 9872us) op_driver.cc:464] REPLICATION: finished
0924 19:57:45.165422 (+ 1106us) write_op.cc:301] APPLY: starting
0924 19:57:45.165482 (+ 60us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:45.165579 (+ 97us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:45.165666 (+ 87us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:45.165840 (+ 174us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:45.165966 (+ 126us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:45.166009 (+ 43us) write_op.cc:312] APPLY: finished
0924 19:57:45.166192 (+ 183us) log.cc:844] Serialized 23 byte log entry
0924 19:57:45.166326 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.166385 (+ 59us) write_op.cc:454] Released schema lock
0924 19:57:45.166415 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:45.166647 (+ 232us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.166673 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":39,"spinlock_wait_cycles":39479040,"child_traces":[["op",{"apply.queue_time_us":179,"apply.run_cpu_time_us":1292,"apply.run_wall_time_us":1291,"num_ops":1,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":2196,"prepare.run_wall_time_us":2197,"raft.queue_time_us":850,"raft.run_cpu_time_us":674,"raft.run_wall_time_us":674,"replication_time_us":14043}],["op",{"apply.queue_time_us":944,"apply.run_cpu_time_us":1192,"apply.run_wall_time_us":1192,"num_ops":1,"prepare.queue_time_us":133,"prepare.run_cpu_time_us":2229,"prepare.run_wall_time_us":22523,"raft.queue_time_us":85,"raft.run_cpu_time_us":649,"raft.run_wall_time_us":650,"replication_time_us":10347}]]}
W20260924 19:57:45.277885 17753 leader_election.cc:343] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.285429 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274f3decdc7e42c0917bf28151c4da71" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:45.380020 17753 leader_election.cc:304] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:45.326848 17800 tablet_service.cc:1511] Processing CreateTablet for tablet 97bbb17c170d42349d2c9cdb41349cb9 (DEFAULT_TABLE table=default.Table_b2824cbbb81c4186840b758c61eccbb2 [id=4e169b185baf4e7a81735a13b6d82fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.380764 18780 raft_consensus.cc:2749] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:45.381203 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97bbb17c170d42349d2c9cdb41349cb9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:45.381548 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 103695us
W20260924 19:57:45.382336 17753 leader_election.cc:343] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.382736 17753 leader_election.cc:304] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:45.383369 18780 raft_consensus.cc:2749] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:45.384325 17753 leader_election.cc:343] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.384723 17753 leader_election.cc:304] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:45.385510 18780 raft_consensus.cc:2749] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:45.380249 17817 raft_consensus.cc:2468] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:45.389602 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 206) took 2403 ms. Trace:
I20260924 19:57:45.390044 18910 rpcz_store.cc:276] 0924 19:57:42.986638 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:42.986756 (+ 118us) service_pool.cc:224] Handling call
0924 19:57:45.356097 (+2369341us) ts_tablet_manager.cc:1151] Deleting tablet 44199d2655f14dcf97afe2180789aae4
0924 19:57:45.372810 (+ 16713us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:45.389537 (+ 16727us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368793}
I20260924 19:57:45.390942 18231 ts_tablet_manager.cc:1929] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:45.391340 18231 log.cc:1199] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/9c861e93ad9f40ba9ad010021e43bc2b
I20260924 19:57:45.392025 18231 ts_tablet_manager.cc:1950] T 9c861e93ad9f40ba9ad010021e43bc2b P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:45.391949 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 212) took 1136 ms. Trace:
I20260924 19:57:45.392462 18910 rpcz_store.cc:276] 0924 19:57:44.255674 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:44.256721 (+ 1047us) service_pool.cc:224] Handling call
0924 19:57:45.391408 (+1134687us) ts_tablet_manager.cc:1151] Deleting tablet 3192f66a492442c1bf4137cfe7e13bc9
0924 19:57:45.391478 (+ 70us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:45.391884 (+ 406us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1134170}
W20260924 19:57:45.393123 17866 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3d00 after lost signal to thread 17715
W20260924 19:57:45.394249 17866 debug-util.cc:398] Leaking SignalData structure 0x7b080006d280 after lost signal to thread 17869
I20260924 19:57:45.395128 19165 raft_consensus.cc:493] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.395601 19165 raft_consensus.cc:515] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:45.397456 19165 leader_election.cc:290] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:45.412847 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274f3decdc7e42c0917bf28151c4da71" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:45.327791 17796 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a524bad1c4040e58ad84fcf5a4fc47d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:37 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.338423 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 201ea0020cb3430eac85e80b8c06fb07 with delete_type TABLET_DATA_DELETED (replaced by tablet a00e3626ea154dc5a959f0fbb4e849dc) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.350886 18224 tablet_bootstrap.cc:492] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:45.423081 18224 ts_tablet_manager.cc:1403] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.245s user 0.012s sys 0.000s
I20260924 19:57:45.396818 17906 leader_election.cc:304] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:45.399390 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f169c47d711d4566b2004cdfd0de725d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:45.425824 18224 raft_consensus.cc:359] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.427086 18224 raft_consensus.cc:385] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:45.427591 18224 raft_consensus.cc:740] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:45.428308 18224 consensus_queue.cc:260] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.430645 18224 ts_tablet_manager.cc:1434] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 19:57:45.431486 18224 tablet_bootstrap.cc:492] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:45.431461 18223 tablet_bootstrap.cc:654] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:45.446336 18224 tablet_bootstrap.cc:654] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:45.456915 18220 tablet_bootstrap.cc:492] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
W20260924 19:57:45.488323 17611 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 33) took 1174 ms (client timeout 1000 ms). Trace:
W20260924 19:57:45.489951 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet ba46ac86b242407abd3e8dbf09f12efa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba46ac86b242407abd3e8dbf09f12efa already in progress: creating tablet
W20260924 19:57:45.458455 18050 leader_election.cc:343] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.460736 18086 tablet_service.cc:1511] Processing CreateTablet for tablet bea1ebdf647e4ef9a131a0398672b8ab (DEFAULT_TABLE table=default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.463975 18091 tablet_service.cc:1511] Processing CreateTablet for tablet 97bbb17c170d42349d2c9cdb41349cb9 (DEFAULT_TABLE table=default.Table_b2824cbbb81c4186840b758c61eccbb2 [id=4e169b185baf4e7a81735a13b6d82fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.482656 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet 201ea0020cb3430eac85e80b8c06fb07 with delete_type TABLET_DATA_DELETED (replaced by tablet a00e3626ea154dc5a959f0fbb4e849dc) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:45.328833 17614 rpcz_store.cc:276] 0924 19:57:44.192183 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.238984 (+1046801us) service_pool.cc:224] Handling call
0924 19:57:45.328385 (+ 89401us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:45.279813 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:45.280217 (+ 404us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:45.280244 (+ 27us) write_op.cc:435] Acquired schema lock
0924 19:57:45.280259 (+ 15us) tablet.cc:662] Decoding operations
0924 19:57:45.280585 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:45.280620 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:45.280646 (+ 26us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:45.280877 (+ 231us) tablet.cc:701] Row locks acquired
0924 19:57:45.280893 (+ 16us) write_op.cc:260] PREPARE: finished
0924 19:57:45.281027 (+ 134us) write_op.cc:270] Start()
0924 19:57:45.281107 (+ 80us) write_op.cc:276] Timestamp: P: 1790279865280939 usec, L: 0
0924 19:57:45.281126 (+ 19us) op_driver.cc:348] REPLICATION: starting
0924 19:57:45.281595 (+ 469us) log.cc:844] Serialized 659 byte log entry
0924 19:57:45.318097 (+ 36502us) op_driver.cc:464] REPLICATION: finished
0924 19:57:45.321261 (+ 3164us) write_op.cc:301] APPLY: starting
0924 19:57:45.321310 (+ 49us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:45.321413 (+ 103us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:45.321510 (+ 97us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:45.321821 (+ 311us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:45.322011 (+ 190us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:45.322055 (+ 44us) write_op.cc:312] APPLY: finished
0924 19:57:45.322210 (+ 155us) log.cc:844] Serialized 29 byte log entry
0924 19:57:45.322320 (+ 110us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.322380 (+ 60us) write_op.cc:454] Released schema lock
0924 19:57:45.322408 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3045,"apply.run_cpu_time_us":1314,"apply.run_wall_time_us":1314,"num_ops":2,"prepare.queue_time_us":3579,"replication_time_us":36878,"threads_started":1}]]}
I20260924 19:57:45.491989 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet ba46ac86b242407abd3e8dbf09f12efa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:45.491492 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]) successfully deleted
W20260924 19:57:45.492906 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet dd2f4eb8bf254c4e85b6a9072271766b because the tablet was not found. No further retry: Not found: Tablet not found: dd2f4eb8bf254c4e85b6a9072271766b
W20260924 19:57:45.493036 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 44199d2655f14dcf97afe2180789aae4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44199d2655f14dcf97afe2180789aae4 already in progress: creating tablet
I20260924 19:57:45.484154 18082 tablet_service.cc:1511] Processing CreateTablet for tablet 10fe8d2d06dd4f83bef8c52a82693344 (DEFAULT_TABLE table=default.Table_fcbe3b81791a4aaf923847b1366eb6f9 [id=25cb87a38f0f473d99bad6f06d4edcf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.394672 17816 raft_consensus.cc:2468] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:45.494787 19180 tablet.cc:1751] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:45.499343 19181 tablet.cc:1751] T edc6c16578e5463a9d44c25fd6c538a9 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260924 19:57:45.504046 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 3192f66a492442c1bf4137cfe7e13bc9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3192f66a492442c1bf4137cfe7e13bc9 already in progress: creating tablet
I20260924 19:57:45.508105 17795 tablet_service.cc:1511] Processing CreateTablet for tablet 10fe8d2d06dd4f83bef8c52a82693344 (DEFAULT_TABLE table=default.Table_fcbe3b81791a4aaf923847b1366eb6f9 [id=25cb87a38f0f473d99bad6f06d4edcf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:45.509142 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fe8d2d06dd4f83bef8c52a82693344. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:45.509883 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 83ee43b6dc064dc7b320691d52e57ea7 because the tablet was not found. No further retry: Not found: Tablet not found: 83ee43b6dc064dc7b320691d52e57ea7
I20260924 19:57:45.501497 19135 raft_consensus.cc:2804] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:57:45.510846 18965 raft_consensus.cc:493] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.511418 18965 raft_consensus.cc:515] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.512590 19135 raft_consensus.cc:493] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election (detected failure of leader eaf64bfb64aa42948c9bc1411ce4ce35)
I20260924 19:57:45.514801 18965 leader_election.cc:290] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:45.516680 19135 raft_consensus.cc:3060] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:57:45.518996 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "484517ee7df8463bb70d201ead1c6835" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:45.523578 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_c42c4d9b6c1e4d2abd6aa51f251be773"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:45.520900 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet f90a677e94034c73ab5452554862c1c3 because the tablet was not found. No further retry: Not found: Tablet not found: f90a677e94034c73ab5452554862c1c3
W20260924 19:57:45.524816 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 83ee43b6dc064dc7b320691d52e57ea7 because the tablet was not found. No further retry: Not found: Tablet not found: 83ee43b6dc064dc7b320691d52e57ea7
W20260924 19:57:45.525336 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 910d61b8c3574302b5a328187226571e because the tablet was not found. No further retry: Not found: Tablet not found: 910d61b8c3574302b5a328187226571e
W20260924 19:57:45.525875 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet ba46ac86b242407abd3e8dbf09f12efa because the tablet was not found. No further retry: Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:57:45.525758 17613 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_818936b597264e41853b635f6232f509" } new_table_name: "default.Table_88b623b6a00845afb22e24e34604e9ea" modify_external_catalogs: true
W20260924 19:57:45.489166 17611 rpcz_store.cc:269] 0924 19:57:44.314360 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:44.314592 (+ 232us) service_pool.cc:224] Handling call
0924 19:57:45.488255 (+1173663us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:45.155660 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:45.156117 (+ 457us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:45.156144 (+ 27us) write_op.cc:435] Acquired schema lock
0924 19:57:45.156163 (+ 19us) tablet.cc:662] Decoding operations
0924 19:57:45.156685 (+ 522us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:45.156723 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:45.156748 (+ 25us) tablet.cc:685] Acquiring locks for 4 operations
0924 19:57:45.157101 (+ 353us) tablet.cc:701] Row locks acquired
0924 19:57:45.157118 (+ 17us) write_op.cc:260] PREPARE: finished
0924 19:57:45.157228 (+ 110us) write_op.cc:270] Start()
0924 19:57:45.157313 (+ 85us) write_op.cc:276] Timestamp: P: 1790279865157165 usec, L: 0
0924 19:57:45.157331 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:45.157850 (+ 519us) log.cc:844] Serialized 1662 byte log entry
0924 19:57:45.176667 (+ 18817us) op_driver.cc:464] REPLICATION: finished
0924 19:57:45.176976 (+ 309us) write_op.cc:301] APPLY: starting
0924 19:57:45.177042 (+ 66us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:45.177183 (+ 141us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:45.177267 (+ 84us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:45.177657 (+ 390us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:45.177801 (+ 144us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0924 19:57:45.177842 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:57:45.178091 (+ 249us) log.cc:844] Serialized 41 byte log entry
0924 19:57:45.178232 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.178309 (+ 77us) write_op.cc:454] Released schema lock
0924 19:57:45.178379 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:45.178603 (+ 224us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.178626 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":173,"apply.run_cpu_time_us":1597,"apply.run_wall_time_us":1597,"num_ops":4,"prepare.queue_time_us":10802,"prepare.run_cpu_time_us":2382,"prepare.run_wall_time_us":2381,"replication_time_us":19232}]]}
I20260924 19:57:45.529304 17611 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_4c4cbf180cb04cf78a5d09a34467e15a" } modify_external_catalogs: true
I20260924 19:57:45.529976 17611 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_4c4cbf180cb04cf78a5d09a34467e15a" } modify_external_catalogs: true
I20260924 19:57:45.534173 18910 tablet_replica.cc:333] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:45.540130 18231 tablet_replica.cc:333] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:45.541900 18231 raft_consensus.cc:2243] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:45.542560 18231 raft_consensus.cc:2272] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:45.544492 18231 ts_tablet_manager.cc:1916] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:45.545578 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 07f86f24d9d14d05898e1c0f69428850 because the tablet was not found. No further retry: Not found: Tablet not found: 07f86f24d9d14d05898e1c0f69428850
W20260924 19:57:45.545673 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c42c4d9b6c1e4d2abd6aa51f251be773 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:45.545949 18910 raft_consensus.cc:2243] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus shutting down.
W20260924 19:57:45.546205 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 44199d2655f14dcf97afe2180789aae4 because the tablet was not found. No further retry: Not found: Tablet not found: 44199d2655f14dcf97afe2180789aae4
I20260924 19:57:45.546389 18910 raft_consensus.cc:2272] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Raft consensus is shut down!
W20260924 19:57:45.546772 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 8a524bad1c4040e58ad84fcf5a4fc47d because the tablet was not found. No further retry: Not found: Tablet not found: 8a524bad1c4040e58ad84fcf5a4fc47d
I20260924 19:57:45.546664 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:45.548041 18910 ts_tablet_manager.cc:1916] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:45.548606 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:45.551749 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_88352246addd43f4be871c983413b2de [id=fa469a44fb3741d39036337129cc36a1]). Expected version 2 got 1
I20260924 19:57:45.573810 18220 tablet_bootstrap.cc:654] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:45.529671 18050 leader_election.cc:343] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.537516 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bea1ebdf647e4ef9a131a0398672b8ab. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:57:45.578280 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 48637.8us
I20260924 19:57:45.578616 18091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97bbb17c170d42349d2c9cdb41349cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.582350 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fe8d2d06dd4f83bef8c52a82693344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:45.600754 18224 tablet_bootstrap.cc:492] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:45.601188 18224 ts_tablet_manager.cc:1403] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.170s user 0.012s sys 0.002s
I20260924 19:57:45.603713 18224 raft_consensus.cc:359] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:45.604293 18224 raft_consensus.cc:385] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:45.604545 18224 raft_consensus.cc:740] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:45.605063 18224 consensus_queue.cc:260] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:45.609319 18224 ts_tablet_manager.cc:1434] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260924 19:57:45.630245 18965 raft_consensus.cc:493] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.630841 18965 raft_consensus.cc:515] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.633563 18224 tablet_bootstrap.cc:492] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:45.634590 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b99ff1920274831b58d8283bd592818" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:45.635249 17817 raft_consensus.cc:2468] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:45.639089 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae7a0660795e4e4b87d0959ca84d5d1b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:45.640151 17601 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet d930e7f4a6b7471899dab1b28e40f93d (table default.Table_9b8f6ad547fc4c3fa379c93590aebd56 [id=4fc8e565a6a640549716cee596fa3f9d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d930e7f4a6b7471899dab1b28e40f93d consensus state has no leader
W20260924 19:57:45.643195 18050 leader_election.cc:343] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.644919 18965 leader_election.cc:290] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:45.646176 18231 ts_tablet_manager.cc:1929] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260924 19:57:45.650395 19051 ts_tablet_manager.cc:1929] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:45.650861 19051 log.cc:1199] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/44199d2655f14dcf97afe2180789aae4
I20260924 19:57:45.656147 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f169c47d711d4566b2004cdfd0de725d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:45.658692 18231 log.cc:1199] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/201ea0020cb3430eac85e80b8c06fb07
I20260924 19:57:45.658825 17821 raft_consensus.cc:2468] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:45.659421 18231 ts_tablet_manager.cc:1950] T 201ea0020cb3430eac85e80b8c06fb07 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:45.661932 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "484517ee7df8463bb70d201ead1c6835" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:45.662561 17815 raft_consensus.cc:2468] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:45.658288 18224 tablet_bootstrap.cc:654] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:45.670979 18051 leader_election.cc:304] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:45.672544 18965 raft_consensus.cc:2804] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:45.673394 18965 raft_consensus.cc:697] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:45.674350 18965 consensus_queue.cc:237] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.675086 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]) successfully deleted
I20260924 19:57:45.681432 18220 tablet_bootstrap.cc:492] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:45.681980 18220 ts_tablet_manager.cc:1403] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.225s user 0.010s sys 0.001s
I20260924 19:57:45.683934 19135 raft_consensus.cc:515] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.687451 18051 leader_election.cc:304] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:45.688284 19165 raft_consensus.cc:2804] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:45.688650 19165 raft_consensus.cc:493] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:45.688994 19165 raft_consensus.cc:3060] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:45.690204 18224 tablet_bootstrap.cc:492] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:45.690653 18224 ts_tablet_manager.cc:1403] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.057s user 0.008s sys 0.002s
I20260924 19:57:45.696214 18224 raft_consensus.cc:359] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.696915 18224 raft_consensus.cc:385] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:45.697190 18224 raft_consensus.cc:740] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:45.698186 19165 raft_consensus.cc:515] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.699077 18780 raft_consensus.cc:493] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.699561 18780 raft_consensus.cc:515] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.700919 19165 leader_election.cc:290] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:45.701612 18780 leader_election.cc:290] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:45.702392 18224 consensus_queue.cc:260] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.706446 18051 leader_election.cc:304] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:45.707391 19167 consensus_queue.cc:1048] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:45.704224 19051 ts_tablet_manager.cc:1950] T 44199d2655f14dcf97afe2180789aae4 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:45.708633 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 205) took 2588 ms. Trace:
I20260924 19:57:45.708715 19167 raft_consensus.cc:2804] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:45.709105 19167 raft_consensus.cc:493] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:45.709388 18224 ts_tablet_manager.cc:1434] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.018s user 0.009s sys 0.000s
I20260924 19:57:45.708945 19051 rpcz_store.cc:276] 0924 19:57:43.120219 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:43.194241 (+ 74022us) service_pool.cc:224] Handling call
0924 19:57:44.974179 (+1779938us) ts_tablet_manager.cc:1151] Deleting tablet 44199d2655f14dcf97afe2180789aae4
0924 19:57:45.018431 (+ 44252us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:45.471837 (+453406us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:45.650256 (+178419us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:45.708579 (+ 58323us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1779232}
I20260924 19:57:45.709501 19167 raft_consensus.cc:3060] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:45.711887 18051 leader_election.cc:304] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:45.713641 19080 raft_consensus.cc:2804] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:45.714036 19080 raft_consensus.cc:493] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:45.714394 19080 raft_consensus.cc:3060] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:45.710407 18224 tablet_bootstrap.cc:492] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:45.701335 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b99ff1920274831b58d8283bd592818" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:45.716167 17966 raft_consensus.cc:1240] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 2 FOLLOWER]: Rejecting Update request from peer eaf64bfb64aa42948c9bc1411ce4ce35 for earlier term 1. Current term is 2. Ops: []
W20260924 19:57:45.717000 18050 leader_election.cc:343] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.720553 18224 tablet_bootstrap.cc:654] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:45.720475 19135 leader_election.cc:290] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:45.720976 19166 consensus_queue.cc:1059] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:45.721915 19166 raft_consensus.cc:3055] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 19:57:45.722213 19166 raft_consensus.cc:740] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:45.723510 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274f3decdc7e42c0917bf28151c4da71" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:45.724543 19167 raft_consensus.cc:515] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:45.726920 18223 tablet_bootstrap.cc:492] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:45.723080 19166 consensus_queue.cc:260] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.728372 18223 ts_tablet_manager.cc:1403] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.487s user 0.013s sys 0.005s
I20260924 19:57:45.728507 19166 raft_consensus.cc:3060] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:57:45.755461 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f169c47d711d4566b2004cdfd0de725d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
W20260924 19:57:45.757905 18050 leader_election.cc:343] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.754503 19080 raft_consensus.cc:515] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.760289 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "274f3decdc7e42c0917bf28151c4da71" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:45.761009 17816 raft_consensus.cc:3060] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:57:45.741665 17607 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:45.793951 18224 tablet_bootstrap.cc:492] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:45.794449 18224 ts_tablet_manager.cc:1403] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.005s
I20260924 19:57:45.797860 18780 raft_consensus.cc:493] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.798316 18780 raft_consensus.cc:515] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.706046 18220 raft_consensus.cc:359] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.799417 18220 raft_consensus.cc:385] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:57:45.799628 17906 consensus_peers.cc:597] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 201ea0020cb3430eac85e80b8c06fb07. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:45.799702 18220 raft_consensus.cc:740] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:45.800302 18780 leader_election.cc:290] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:45.800736 18780 raft_consensus.cc:493] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.800341 18220 consensus_queue.cc:260] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.801213 18780 raft_consensus.cc:515] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.802492 18220 ts_tablet_manager.cc:1434] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.120s user 0.006s sys 0.000s
I20260924 19:57:45.805258 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d129e0979a584f40ba9a975349cf15a5" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:45.807367 18780 leader_election.cc:290] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:45.797024 18224 raft_consensus.cc:359] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.808274 18224 raft_consensus.cc:385] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:45.808765 18224 raft_consensus.cc:740] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:45.809661 18224 consensus_queue.cc:260] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.820020 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b99ff1920274831b58d8283bd592818" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:45.820425 18224 ts_tablet_manager.cc:1434] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.026s user 0.007s sys 0.001s
I20260924 19:57:45.820780 17817 raft_consensus.cc:3060] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:45.830672 18910 ts_tablet_manager.cc:1929] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:45.840166 17612 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 251) took 1612 ms. Trace:
I20260924 19:57:45.843784 17615 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet c058a49bfd5544b58b76edfd3fadbd1a
I20260924 19:57:45.843428 17611 catalog_manager.cc:6128] Not sending DeleteTablet requests; no consensus state for tablet f549849add4345fca4ff47e155650736
I20260924 19:57:45.846837 17610 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 5f116f866c2b4674b4eef531cf00af66
I20260924 19:57:45.849959 17606 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 249) took 1922 ms. Trace:
W20260924 19:57:45.843392 17609 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 43) took 1234 ms (client timeout 1000 ms). Trace:
W20260924 19:57:45.851325 17609 rpcz_store.cc:269] 0924 19:57:44.608984 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.109414 (+500430us) service_pool.cc:224] Handling call
0924 19:57:45.843315 (+733901us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":176292,"reported_tablets":27}
I20260924 19:57:45.854571 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f116f866c2b4674b4eef531cf00af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:45 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.840538 17612 rpcz_store.cc:276] 0924 19:57:44.227668 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.259118 (+1031450us) service_pool.cc:224] Handling call
0924 19:57:45.301878 (+ 42760us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:45.303464 (+ 1586us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:45.576166 (+272702us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:45.839782 (+263616us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:45.840099 (+ 317us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:45.532441 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:45.532934 (+ 493us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:45.532972 (+ 38us) write_op.cc:435] Acquired schema lock
0924 19:57:45.532991 (+ 19us) tablet.cc:662] Decoding operations
0924 19:57:45.533283 (+ 292us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:45.533324 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:45.533348 (+ 24us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:45.533571 (+ 223us) tablet.cc:701] Row locks acquired
0924 19:57:45.533584 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:57:45.533718 (+ 134us) write_op.cc:270] Start()
0924 19:57:45.533812 (+ 94us) write_op.cc:276] Timestamp: P: 1790279865533662 usec, L: 0
0924 19:57:45.533830 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:45.534308 (+ 478us) log.cc:844] Serialized 433 byte log entry
0924 19:57:45.571887 (+ 37579us) op_driver.cc:464] REPLICATION: finished
0924 19:57:45.574413 (+ 2526us) write_op.cc:301] APPLY: starting
0924 19:57:45.574464 (+ 51us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:45.574568 (+ 104us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:45.574675 (+ 107us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:45.575098 (+ 423us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:45.575576 (+ 478us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:45.575617 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:57:45.575774 (+ 157us) log.cc:844] Serialized 29 byte log entry
0924 19:57:45.575896 (+ 122us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.575955 (+ 59us) write_op.cc:454] Released schema lock
0924 19:57:45.575994 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:45.576511 (+ 517us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.576534 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2371,"apply.run_cpu_time_us":1510,"apply.run_wall_time_us":2091,"num_ops":2,"prepare.queue_time_us":227717,"prepare.run_cpu_time_us":2169,"prepare.run_wall_time_us":31874,"raft.queue_time_us":29665,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":13,"replication_time_us":37975}]]}
I20260924 19:57:45.855463 17612 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_bda24aa05e4c497385d8a40f8c9752b4" } new_table_name: "default.Table_db4a5e39fd954bb0b4c271899feefe90" modify_external_catalogs: true
W20260924 19:57:45.858723 17753 leader_election.cc:343] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.821427 18224 tablet_bootstrap.cc:492] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:45.826417 17816 raft_consensus.cc:2468] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 2.
I20260924 19:57:45.844666 17872 heartbeater.cc:344] Connected to a master server at 127.17.2.190:46309
I20260924 19:57:45.846527 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet c058a49bfd5544b58b76edfd3fadbd1a with delete_type TABLET_DATA_DELETED (replaced by tablet 46e703a4892f445bb0a1c9f5a46db63d) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:45.832706 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89646893588340038f30ed4754623ec2" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:45.841312 17941 tablet_service.cc:1558] Processing DeleteTablet for tablet d970cc92a1134a98a613d198a319cb01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:32 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:45.844857 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bafaab10ce564b1b97659eccc0ab0967" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:45.845839 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet c058a49bfd5544b58b76edfd3fadbd1a with delete_type TABLET_DATA_DELETED (replaced by tablet 46e703a4892f445bb0a1c9f5a46db63d) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:45.880581 19080 leader_election.cc:290] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:45.848511 17945 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f116f866c2b4674b4eef531cf00af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:45 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:45.882678 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bafaab10ce564b1b97659eccc0ab0967" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:45.853699 18220 tablet_bootstrap.cc:492] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:45.860596 17906 leader_election.cc:304] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:45.888748 19170 raft_consensus.cc:2804] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:57:45.889210 19170 raft_consensus.cc:697] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [term 2 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:45.890013 19170 consensus_queue.cc:237] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.890551 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "484517ee7df8463bb70d201ead1c6835" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
W20260924 19:57:45.892227 18050 leader_election.cc:343] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:45.792138 19167 leader_election.cc:290] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:45.794770 19165 raft_consensus.cc:493] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:45.795529 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 83725.3us
I20260924 19:57:45.872259 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d129e0979a584f40ba9a975349cf15a5" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:45.872970 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89646893588340038f30ed4754623ec2" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:45.899430 18051 leader_election.cc:343] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: ae7a0660795e4e4b87d0959ca84d5d1b
I20260924 19:57:45.903043 18910 log.cc:1199] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/fec9b6ecfb384b528cf71c4cfb2fd4cb
I20260924 19:57:45.903416 19165 raft_consensus.cc:515] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.906219 18224 tablet_bootstrap.cc:654] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:45.899788 18051 leader_election.cc:304] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:45.911847 18910 ts_tablet_manager.cc:1950] T fec9b6ecfb384b528cf71c4cfb2fd4cb P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:45.916049 19165 leader_election.cc:290] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:45.919538 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431d298f513d47268284d50af2ea1436" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:45.857512 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 5f116f866c2b4674b4eef531cf00af66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f116f866c2b4674b4eef531cf00af66 already in progress: creating tablet
I20260924 19:57:45.846292 17615 catalog_manager.cc:6934] ReplaceTablet: tablet c058a49bfd5544b58b76edfd3fadbd1a deleted and replaced by tablet 46e703a4892f445bb0a1c9f5a46db63d
I20260924 19:57:45.926093 17615 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51280 (request call id 254) took 1403 ms. Trace:
W20260924 19:57:45.926026 17600 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 68544.5us
W20260924 19:57:45.926899 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet c058a49bfd5544b58b76edfd3fadbd1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c058a49bfd5544b58b76edfd3fadbd1a already in progress: creating tablet
I20260924 19:57:45.927784 19165 raft_consensus.cc:2749] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:45.926406 17615 rpcz_store.cc:276] 0924 19:57:44.523217 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.334357 (+811140us) service_pool.cc:224] Handling call
0924 19:57:45.536646 (+202289us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552209ce1 00007fa54fdd3d37 00007fa54fdd3cc2 00007fa54fdd3bf1 00007fa54fdd3b6c 00007fa54fdd2e22 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:45.926032 (+389386us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:45.635349 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:45.734674 (+ 99325us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:45.734710 (+ 36us) write_op.cc:435] Acquired schema lock
0924 19:57:45.734729 (+ 19us) tablet.cc:662] Decoding operations
0924 19:57:45.735128 (+ 399us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:45.735193 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:45.735220 (+ 27us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:45.735450 (+ 230us) tablet.cc:701] Row locks acquired
0924 19:57:45.735466 (+ 16us) write_op.cc:260] PREPARE: finished
0924 19:57:45.735593 (+ 127us) write_op.cc:270] Start()
0924 19:57:45.735682 (+ 89us) write_op.cc:276] Timestamp: P: 1790279865735517 usec, L: 0
0924 19:57:45.735700 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:45.736241 (+ 541us) log.cc:844] Serialized 625 byte log entry
0924 19:57:45.745269 (+ 9028us) op_driver.cc:464] REPLICATION: finished
0924 19:57:45.747067 (+ 1798us) write_op.cc:301] APPLY: starting
0924 19:57:45.747132 (+ 65us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:45.747736 (+ 604us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:45.747851 (+ 115us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:45.748271 (+ 420us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:45.748486 (+ 215us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:45.748535 (+ 49us) write_op.cc:312] APPLY: finished
0924 19:57:45.748716 (+ 181us) log.cc:844] Serialized 29 byte log entry
0924 19:57:45.748862 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.749106 (+ 244us) write_op.cc:454] Released schema lock
0924 19:57:45.749142 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:45.750079 (+ 937us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.750132 (+ 53us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":462599552,"child_traces":[["op",{"apply.queue_time_us":1632,"apply.run_cpu_time_us":1747,"apply.run_wall_time_us":2921,"num_ops":2,"prepare.queue_time_us":95970,"prepare.run_cpu_time_us":2592,"prepare.run_wall_time_us":101380,"raft.queue_time_us":1036,"raft.run_cpu_time_us":744,"raft.run_wall_time_us":2947,"replication_time_us":9465}]]}
I20260924 19:57:45.930512 17615 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_b8cbd748636249c0ab17657b8f259893"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:45.931020 18050 leader_election.cc:343] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:45.933205 17615 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8cbd748636249c0ab17657b8f259893 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:45.935834 17967 raft_consensus.cc:3060] T 92c0daa7d67e4aa390e334c669adda76 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:45.936756 19135 raft_consensus.cc:493] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:45.937171 19135 raft_consensus.cc:515] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.941025 19135 leader_election.cc:290] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:45.944248 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 210) took 1821 ms. Trace:
I20260924 19:57:45.944895 17817 raft_consensus.cc:2468] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:45.944734 19051 rpcz_store.cc:276] 0924 19:57:44.122845 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:44.128271 (+ 5426us) service_pool.cc:224] Handling call
0924 19:57:45.943671 (+1815400us) ts_tablet_manager.cc:1151] Deleting tablet 3192f66a492442c1bf4137cfe7e13bc9
0924 19:57:45.943748 (+ 77us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:45.944159 (+ 411us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806812}
I20260924 19:57:45.946955 19051 tablet_replica.cc:333] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:45.911046 18111 raft_consensus.cc:2468] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 2.
I20260924 19:57:45.947999 19051 raft_consensus.cc:2243] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:45.950224 18223 raft_consensus.cc:359] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.951056 18223 raft_consensus.cc:385] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:45.951469 18223 raft_consensus.cc:740] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:45.952384 18223 consensus_queue.cc:260] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:45.954435 18220 tablet_bootstrap.cc:654] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:45.954849 18223 ts_tablet_manager.cc:1434] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.226s user 0.005s sys 0.001s
I20260924 19:57:45.859370 17610 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 252) took 1553 ms. Trace:
I20260924 19:57:45.890131 17608 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 253) took 1373 ms. Trace:
I20260924 19:57:45.957875 18082 tablet_service.cc:1558] Processing DeleteTablet for tablet c058a49bfd5544b58b76edfd3fadbd1a with delete_type TABLET_DATA_DELETED (replaced by tablet 46e703a4892f445bb0a1c9f5a46db63d) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:45.955441 17608 rpcz_store.cc:276] 0924 19:57:44.516879 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.333552 (+816673us) service_pool.cc:224] Handling call
0924 19:57:45.523468 (+189916us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552207166 00007fa54fdb3034 00007fa54fdb2fc2 00007fa54fdb2ef1 00007fa54fdb2e6c 00007fa54fdb2122 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:45.829653 (+306185us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:45.836361 (+ 6708us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:45.886259 (+ 49898us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:45.889219 (+ 2960us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:45.890066 (+ 847us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:45.856172 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:45.869276 (+ 13104us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:45.869302 (+ 26us) write_op.cc:435] Acquired schema lock
0924 19:57:45.869317 (+ 15us) tablet.cc:662] Decoding operations
0924 19:57:45.869599 (+ 282us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:45.869640 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:45.869665 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:45.869874 (+ 209us) tablet.cc:701] Row locks acquired
0924 19:57:45.869889 (+ 15us) write_op.cc:260] PREPARE: finished
0924 19:57:45.870028 (+ 139us) write_op.cc:270] Start()
0924 19:57:45.870114 (+ 86us) write_op.cc:276] Timestamp: P: 1790279865869943 usec, L: 0
0924 19:57:45.870129 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:57:45.870629 (+ 500us) log.cc:844] Serialized 433 byte log entry
0924 19:57:45.880741 (+ 10112us) op_driver.cc:464] REPLICATION: finished
0924 19:57:45.881124 (+ 383us) write_op.cc:301] APPLY: starting
0924 19:57:45.881189 (+ 65us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:45.881304 (+ 115us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:45.881421 (+ 117us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:45.881830 (+ 409us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:45.881987 (+ 157us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:45.882035 (+ 48us) write_op.cc:312] APPLY: finished
0924 19:57:45.882227 (+ 192us) log.cc:844] Serialized 29 byte log entry
0924 19:57:45.882378 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.882451 (+ 73us) write_op.cc:454] Released schema lock
0924 19:57:45.882485 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:45.882715 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.882748 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":434993536,"child_traces":[["op",{"apply.queue_time_us":227,"apply.run_cpu_time_us":1567,"apply.run_wall_time_us":1568,"num_ops":2,"prepare.queue_time_us":18564,"prepare.run_cpu_time_us":2096,"prepare.run_wall_time_us":14745,"raft.queue_time_us":1004,"raft.run_cpu_time_us":714,"raft.run_wall_time_us":715,"replication_time_us":10514}]]}
I20260924 19:57:45.913133 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet fec9b6ecfb384b528cf71c4cfb2fd4cb (table default.Table_412220a60c3e4a16a5c4abcbbc48357c [id=f3d6301471034414b8cc2892f7b8a885]) successfully deleted
I20260924 19:57:45.958832 18082 tablet_service.cc:1558] Processing DeleteTablet for tablet 5f116f866c2b4674b4eef531cf00af66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:45 UTC) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:45.959223 17599 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 46111.9us
I20260924 19:57:45.959820 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 44199d2655f14dcf97afe2180789aae4 (table default.Table_1b2f1b739e234953861c274a098c2e1c [id=f02f0018bf7346c6a0ba91d52909be98]) successfully deleted
W20260924 19:57:45.960502 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 3192f66a492442c1bf4137cfe7e13bc9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3192f66a492442c1bf4137cfe7e13bc9 already in progress: creating tablet
W20260924 19:57:45.961113 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 910d61b8c3574302b5a328187226571e because the tablet was not found. No further retry: Not found: Tablet not found: 910d61b8c3574302b5a328187226571e
W20260924 19:57:45.961733 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 910d61b8c3574302b5a328187226571e because the tablet was not found. No further retry: Not found: Tablet not found: 910d61b8c3574302b5a328187226571e
W20260924 19:57:45.962496 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet dd2f4eb8bf254c4e85b6a9072271766b because the tablet was not found. No further retry: Not found: Tablet not found: dd2f4eb8bf254c4e85b6a9072271766b
W20260924 19:57:45.964314 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet f90a677e94034c73ab5452554862c1c3 because the tablet was not found. No further retry: Not found: Tablet not found: f90a677e94034c73ab5452554862c1c3
W20260924 19:57:45.965047 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet a765abd702644f079bddffb3fc4238ef because the tablet was not found. No further retry: Not found: Tablet not found: a765abd702644f079bddffb3fc4238ef
W20260924 19:57:45.965744 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 8a524bad1c4040e58ad84fcf5a4fc47d because the tablet was not found. No further retry: Not found: Tablet not found: 8a524bad1c4040e58ad84fcf5a4fc47d
I20260924 19:57:45.967994 19051 raft_consensus.cc:2272] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:45.850358 17606 rpcz_store.cc:276] 0924 19:57:43.927246 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.237757 (+1310511us) service_pool.cc:224] Handling call
0924 19:57:45.285248 (+ 47491us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:57:45.286660 (+ 1412us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:57:45.575277 (+288617us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:57:45.849642 (+274365us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:57:45.849904 (+ 262us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:45.340364 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:45.340848 (+ 484us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:45.340876 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:57:45.340892 (+ 16us) tablet.cc:662] Decoding operations
0924 19:57:45.341227 (+ 335us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:45.341274 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:45.341298 (+ 24us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:45.341525 (+ 227us) tablet.cc:701] Row locks acquired
0924 19:57:45.341538 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:57:45.341652 (+ 114us) write_op.cc:270] Start()
0924 19:57:45.341766 (+ 114us) write_op.cc:276] Timestamp: P: 1790279865341573 usec, L: 0
0924 19:57:45.341788 (+ 22us) op_driver.cc:348] REPLICATION: starting
0924 19:57:45.530338 (+188550us) log.cc:844] Serialized 433 byte log entry
0924 19:57:45.532107 (+ 1769us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f7aa1 00007fa55038a5e8 00007fa55081a741 00007fa550819f18 00007fa55081ebb1 00007fa55081eb69 00007fa55081eaf9 00007fa55081eac1 00007fa55081dced 00000000005bd4e4 00000000005bd319
0924 19:57:45.571658 (+ 39551us) op_driver.cc:464] REPLICATION: finished
0924 19:57:45.572745 (+ 1087us) write_op.cc:301] APPLY: starting
0924 19:57:45.572800 (+ 55us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:45.572900 (+ 100us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:45.572919 (+ 19us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:45.573306 (+ 387us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:45.573436 (+ 130us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:57:45.573476 (+ 40us) write_op.cc:312] APPLY: finished
0924 19:57:45.573657 (+ 181us) log.cc:844] Serialized 29 byte log entry
0924 19:57:45.573814 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.573875 (+ 61us) write_op.cc:454] Released schema lock
0924 19:57:45.573905 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:45.574137 (+ 232us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.574159 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":946,"apply.run_cpu_time_us":1363,"apply.run_wall_time_us":1362,"num_ops":2,"prepare.queue_time_us":52545,"prepare.run_cpu_time_us":5161,"prepare.run_wall_time_us":191938,"raft.queue_time_us":30959,"raft.run_cpu_time_us":829,"raft.run_wall_time_us":830,"replication_time_us":229766,"spinlock_wait_cycles":432365440}]]}
I20260924 19:57:45.925276 17607 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:45.973377 17607 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 2e6d926ff7c24901a15b0c7b2239fbad (table default.Table_7c27cdce0a3b40f69142de52aaf39493 [id=65f397bcbc164d0caa6cecae1bab74da]). Expected version 1 got 0
I20260924 19:57:45.975631 17607 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet f43f2a255c644ca6a68a8d867798fc1a (table default.Table_4e3a3158cde24302806b7ba7827831c4 [id=fa0b10df171340868b0305a28d192ea5]). Expected version 1 got 0
I20260924 19:57:45.980706 17607 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 3 got 0
I20260924 19:57:45.982336 17607 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_88352246addd43f4be871c983413b2de [id=fa469a44fb3741d39036337129cc36a1]). Expected version 2 got 1
I20260924 19:57:45.983786 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e6d926ff7c24901a15b0c7b2239fbad" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:45.985873 18223 tablet_bootstrap.cc:492] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:45.979598 18910 tablet_replica.cc:333] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:45.997591 18910 raft_consensus.cc:2243] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:45.998075 18910 raft_consensus.cc:2272] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:46.001426 18910 ts_tablet_manager.cc:1916] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:46.002406 19051 ts_tablet_manager.cc:1916] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:46.013038 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 94924.9us
W20260924 19:57:46.013710 19147 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 15076429791993244055) tablet 00000000000000000000000000000000 cur holder 0x7b5800061500
I20260924 19:57:46.015316 18223 tablet_bootstrap.cc:654] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:46.018435 18220 tablet_bootstrap.cc:492] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:46.018864 18220 ts_tablet_manager.cc:1403] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.165s user 0.010s sys 0.000s
I20260924 19:57:46.019541 19180 tablet.cc:1751] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:46.022260 18220 raft_consensus.cc:359] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.022889 18220 raft_consensus.cc:385] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.027861 19051 ts_tablet_manager.cc:1929] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:46.032469 19051 log.cc:1199] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/07f86f24d9d14d05898e1c0f69428850
I20260924 19:57:46.036417 19051 ts_tablet_manager.cc:1950] T 07f86f24d9d14d05898e1c0f69428850 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:46.039939 19051 tablet_replica.cc:333] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:46.047977 18224 tablet_bootstrap.cc:492] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:46.048462 18224 ts_tablet_manager.cc:1403] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.227s user 0.013s sys 0.006s
I20260924 19:57:46.050766 19051 raft_consensus.cc:2243] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:46.052167 17607 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]). Expected version 1 got 0
I20260924 19:57:46.053179 19051 raft_consensus.cc:2272] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:46.043068 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 07f86f24d9d14d05898e1c0f69428850 (table default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]) successfully deleted
W20260924 19:57:46.055711 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet a765abd702644f079bddffb3fc4238ef because the tablet was not found. No further retry: Not found: Tablet not found: a765abd702644f079bddffb3fc4238ef
I20260924 19:57:46.061196 18220 raft_consensus.cc:740] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.064656 19051 ts_tablet_manager.cc:1916] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:46.067659 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_b7e4d07630f24ee0ab58d5269687448d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:46.069617 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7e4d07630f24ee0ab58d5269687448d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:45.966492 17610 rpcz_store.cc:276] 0924 19:57:44.305635 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.284779 (+979144us) service_pool.cc:224] Handling call
0924 19:57:45.315493 (+ 30714us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552207476 00007fa54fdb7b34 00007fa54fdb7ac2 00007fa54fdb79f1 00007fa54fdb796c 00007fa54fdb6c22 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:45.319393 (+ 3900us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:45.319867 (+ 474us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:45.319994 (+ 127us) catalog_manager.cc:2863] Locking tablets
0924 19:57:45.320500 (+ 506us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:45.602070 (+281570us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:45.846663 (+244593us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:45.846755 (+ 92us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:45.859313 (+ 12558us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:45.564425 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:45.564819 (+ 394us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:45.564842 (+ 23us) write_op.cc:435] Acquired schema lock
0924 19:57:45.564856 (+ 14us) tablet.cc:662] Decoding operations
0924 19:57:45.565168 (+ 312us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:45.565202 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:45.565224 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:45.566140 (+ 916us) tablet.cc:701] Row locks acquired
0924 19:57:45.566157 (+ 17us) write_op.cc:260] PREPARE: finished
0924 19:57:45.566339 (+ 182us) write_op.cc:270] Start()
0924 19:57:45.566462 (+ 123us) write_op.cc:276] Timestamp: P: 1790279865566259 usec, L: 0
0924 19:57:45.566478 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:45.570195 (+ 3717us) log.cc:844] Serialized 728 byte log entry
0924 19:57:45.599827 (+ 29632us) op_driver.cc:464] REPLICATION: finished
0924 19:57:45.600169 (+ 342us) write_op.cc:301] APPLY: starting
0924 19:57:45.600233 (+ 64us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:45.600337 (+ 104us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:45.600445 (+ 108us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:45.600790 (+ 345us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:45.600946 (+ 156us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:45.601092 (+ 146us) write_op.cc:312] APPLY: finished
0924 19:57:45.601380 (+ 288us) log.cc:844] Serialized 29 byte log entry
0924 19:57:45.601526 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.601670 (+ 144us) write_op.cc:454] Released schema lock
0924 19:57:45.601709 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:45.601959 (+ 250us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.601989 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":68916096,"child_traces":[["op",{"apply.queue_time_us":182,"apply.run_cpu_time_us":1454,"apply.run_wall_time_us":1762,"num_ops":2,"prepare.queue_time_us":242281,"prepare.run_cpu_time_us":2295,"prepare.run_wall_time_us":24845,"replication_time_us":33233}]]}
I20260924 19:57:46.049758 17615 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba" } modify_external_catalogs: true
I20260924 19:57:46.076889 17615 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba" } modify_external_catalogs: true
I20260924 19:57:46.064061 18220 consensus_queue.cc:260] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.092360 19135 raft_consensus.cc:493] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:46.092813 19135 raft_consensus.cc:515] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.094748 19135 leader_election.cc:290] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:46.097360 17905 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:57:46.106164 18220 ts_tablet_manager.cc:1434] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.087s user 0.006s sys 0.000s
I20260924 19:57:46.107082 18220 tablet_bootstrap.cc:492] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:46.109964 17966 raft_consensus.cc:3060] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:46.124286 18223 tablet_bootstrap.cc:492] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:46.124852 18223 ts_tablet_manager.cc:1403] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.139s user 0.015s sys 0.012s
I20260924 19:57:46.127475 18223 raft_consensus.cc:359] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.130410 18220 tablet_bootstrap.cc:654] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:46.131214 18019 heartbeater.cc:344] Connected to a master server at 127.17.2.190:46309
I20260924 19:57:46.140121 18223 raft_consensus.cc:385] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.140585 18223 raft_consensus.cc:740] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.148607 18223 consensus_queue.cc:260] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.154273 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f169c47d711d4566b2004cdfd0de725d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:46.156042 17815 raft_consensus.cc:3060] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:46.157725 17818 raft_consensus.cc:1240] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 2 FOLLOWER]: Rejecting Update request from peer eaf64bfb64aa42948c9bc1411ce4ce35 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260924 19:57:46.119087 17614 catalog_manager.cc:5838] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:46.164928 17614 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet f43f2a255c644ca6a68a8d867798fc1a (table default.Table_4e3a3158cde24302806b7ba7827831c4 [id=fa0b10df171340868b0305a28d192ea5]). Expected version 1 got 0
I20260924 19:57:46.168601 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_9faf28b24a2c47449a862536ea873f36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:46.162240 17814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431d298f513d47268284d50af2ea1436" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:46.170151 17814 raft_consensus.cc:2468] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
W20260924 19:57:46.173377 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9faf28b24a2c47449a862536ea873f36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:46.173734 19181 tablet.cc:1751] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:46.166839 17614 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:46.181744 18223 ts_tablet_manager.cc:1434] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.057s user 0.005s sys 0.000s
I20260924 19:57:46.183851 17610 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_0a22025767fb456485c2624d5ca0dcbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:46.185355 17610 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a22025767fb456485c2624d5ca0dcbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:46.188946 17822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e6d926ff7c24901a15b0c7b2239fbad" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:46.193992 18910 ts_tablet_manager.cc:1929] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:46.194342 18910 log.cc:1199] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/9c861e93ad9f40ba9ad010021e43bc2b
I20260924 19:57:46.204664 18224 raft_consensus.cc:359] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.205617 18224 raft_consensus.cc:385] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.205948 18224 raft_consensus.cc:740] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.209774 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_dea06619a5364df0961e47cb617c9e52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:46.211316 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dea06619a5364df0961e47cb617c9e52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:46.215049 18223 tablet_bootstrap.cc:492] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:46.195057 18910 ts_tablet_manager.cc:1950] T 9c861e93ad9f40ba9ad010021e43bc2b P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:46.221048 18223 tablet_bootstrap.cc:654] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:46.222497 18050 consensus_peers.cc:597] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:46.206677 18224 consensus_queue.cc:260] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.225723 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "484517ee7df8463bb70d201ead1c6835" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:46.226383 17823 raft_consensus.cc:3060] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:46.233456 18780 raft_consensus.cc:493] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:46.234222 18780 raft_consensus.cc:515] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:46.236819 17753 leader_election.cc:343] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:46.237749 17753 leader_election.cc:343] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:46.238417 17753 leader_election.cc:343] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:46.245211 19051 ts_tablet_manager.cc:1929] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:46.247469 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 227) took 1146 ms. Trace:
I20260924 19:57:46.247787 18910 rpcz_store.cc:276] 0924 19:57:45.101092 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.101215 (+ 123us) service_pool.cc:224] Handling call
0924 19:57:45.979432 (+878217us) ts_tablet_manager.cc:1151] Deleting tablet 9c861e93ad9f40ba9ad010021e43bc2b
0924 19:57:45.979486 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:46.078163 (+ 98677us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:46.193860 (+115697us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:46.247394 (+ 53534us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":771611}
I20260924 19:57:46.249127 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]) successfully deleted
I20260924 19:57:46.253566 17823 raft_consensus.cc:2468] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:46.268932 18220 tablet_bootstrap.cc:492] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:46.268999 17906 leader_election.cc:304] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:46.269351 18220 ts_tablet_manager.cc:1403] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.162s user 0.006s sys 0.003s
I20260924 19:57:46.270109 19170 raft_consensus.cc:2749] T 2e6d926ff7c24901a15b0c7b2239fbad P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:46.271214 17906 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:57:46.271600 17906 leader_election.cc:304] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:46.272614 19170 raft_consensus.cc:2749] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:46.273623 19051 log.cc:1199] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/9c861e93ad9f40ba9ad010021e43bc2b
I20260924 19:57:46.274091 18220 raft_consensus.cc:359] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.274726 18220 raft_consensus.cc:385] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.275336 18910 tablet_replica.cc:333] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:46.276052 18910 raft_consensus.cc:2243] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:46.276383 19051 ts_tablet_manager.cc:1950] T 9c861e93ad9f40ba9ad010021e43bc2b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:46.276512 18910 raft_consensus.cc:2272] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:46.278194 18910 ts_tablet_manager.cc:1916] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:46.285274 17815 raft_consensus.cc:2468] T f169c47d711d4566b2004cdfd0de725d P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:46.274972 18220 raft_consensus.cc:740] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.287628 18220 consensus_queue.cc:260] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.289606 18220 ts_tablet_manager.cc:1434] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20260924 19:57:46.290200 17606 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_8be4bd6989c649469efa4d12976b4576" } modify_external_catalogs: true
I20260924 19:57:46.292424 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 226) took 1054 ms. Trace:
I20260924 19:57:46.292785 19051 rpcz_store.cc:276] 0924 19:57:45.237906 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.276105 (+ 38199us) service_pool.cc:224] Handling call
0924 19:57:46.039793 (+763688us) ts_tablet_manager.cc:1151] Deleting tablet 9c861e93ad9f40ba9ad010021e43bc2b
0924 19:57:46.039831 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:46.139808 (+ 99977us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:46.245055 (+105247us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:46.292349 (+ 47294us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":763128}
I20260924 19:57:46.293943 17606 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_8be4bd6989c649469efa4d12976b4576" } modify_external_catalogs: true
I20260924 19:57:46.294301 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 9c861e93ad9f40ba9ad010021e43bc2b (table default.Table_d944ee03376e4f54ae842e8c7dfb5b5c [id=46e9e36bd86e4fd4b48aee53df748e9c]) successfully deleted
I20260924 19:57:46.299873 19051 tablet_replica.cc:333] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:46.300776 19051 raft_consensus.cc:2243] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:46.303514 19051 raft_consensus.cc:2272] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:46.292074 17614 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 2e6d926ff7c24901a15b0c7b2239fbad (table default.Table_7c27cdce0a3b40f69142de52aaf39493 [id=65f397bcbc164d0caa6cecae1bab74da]). Expected version 1 got 0
I20260924 19:57:46.307926 18220 tablet_bootstrap.cc:492] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:46.325098 17614 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]). Expected version 1 got 0
I20260924 19:57:46.313148 19051 ts_tablet_manager.cc:1916] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:46.329676 18220 tablet_bootstrap.cc:654] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:46.331243 17614 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:46.238790 17753 leader_election.cc:304] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:46.257263 18780 leader_election.cc:290] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:46.332872 18780 raft_consensus.cc:2749] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:46.333071 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 94683.8us
W20260924 19:57:46.334025 17753 leader_election.cc:343] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:46.334386 17753 leader_election.cc:304] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:46.335011 18780 raft_consensus.cc:2749] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:46.335358 17753 leader_election.cc:343] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:46.335737 17753 leader_election.cc:304] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:46.336371 18780 raft_consensus.cc:2749] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:46.337760 18051 leader_election.cc:304] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:46.341741 18051 leader_election.cc:304] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:46.342978 19080 raft_consensus.cc:2804] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:46.343324 19080 raft_consensus.cc:493] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:46.343631 19080 raft_consensus.cc:3060] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:46.344969 17606 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet cb111f7a189c4f00a41686d647c64e65
I20260924 19:57:46.253930 18224 ts_tablet_manager.cc:1434] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.205s user 0.008s sys 0.000s
I20260924 19:57:46.346726 17945 tablet_service.cc:1558] Processing DeleteTablet for tablet cb111f7a189c4f00a41686d647c64e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:46 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:46.348402 19203 raft_consensus.cc:2804] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:46.348177 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_5128840101ca460b9e13dfd054764e9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:46.348856 19203 raft_consensus.cc:697] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
W20260924 19:57:46.349715 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5128840101ca460b9e13dfd054764e9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:46.349570 19203 consensus_queue.cc:237] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.351298 18224 tablet_bootstrap.cc:492] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:46.358848 19204 consensus_queue.cc:1048] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:46.359640 18223 tablet_bootstrap.cc:492] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:46.360210 18223 ts_tablet_manager.cc:1403] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.178s user 0.007s sys 0.006s
I20260924 19:57:46.317677 19215 tablet.cc:1751] T edc6c16578e5463a9d44c25fd6c538a9 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:46.324147 19212 tablet.cc:1751] T 2e6d926ff7c24901a15b0c7b2239fbad P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:46.368670 18082 tablet_service.cc:1558] Processing DeleteTablet for tablet cb111f7a189c4f00a41686d647c64e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:46 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:46.368819 18224 tablet_bootstrap.cc:654] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:46.372866 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet cb111f7a189c4f00a41686d647c64e65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:46 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:46.376302 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aef0359a77240679240823612507ee8" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:46.376479 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aef0359a77240679240823612507ee8" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:46.377055 18111 raft_consensus.cc:2468] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:46.378536 17614 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_88352246addd43f4be871c983413b2de [id=fa469a44fb3741d39036337129cc36a1]). Expected version 2 got 1
I20260924 19:57:46.381297 18220 tablet_bootstrap.cc:492] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:46.381711 18220 ts_tablet_manager.cc:1403] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.006s
I20260924 19:57:46.383363 18051 leader_election.cc:304] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:46.384298 19204 raft_consensus.cc:2804] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:46.384182 18220 raft_consensus.cc:359] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.384855 18220 raft_consensus.cc:385] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.385133 18220 raft_consensus.cc:740] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.385232 19204 raft_consensus.cc:697] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:46.385749 18220 consensus_queue.cc:260] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.386459 19204 consensus_queue.cc:237] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.390810 18910 ts_tablet_manager.cc:1929] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:46.391212 18910 log.cc:1199] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/07f86f24d9d14d05898e1c0f69428850
I20260924 19:57:46.401129 17614 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 3 got 0
I20260924 19:57:46.407521 18220 ts_tablet_manager.cc:1434] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260924 19:57:46.407717 18910 ts_tablet_manager.cc:1950] T 07f86f24d9d14d05898e1c0f69428850 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:46.408499 18220 tablet_bootstrap.cc:492] T f169c47d711d4566b2004cdfd0de725d P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:46.417456 18223 raft_consensus.cc:359] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.418150 18223 raft_consensus.cc:385] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.418421 18223 raft_consensus.cc:740] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.422324 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 236) took 1211 ms. Trace:
I20260924 19:57:46.422641 18910 rpcz_store.cc:276] 0924 19:57:45.210832 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.210948 (+ 116us) service_pool.cc:224] Handling call
0924 19:57:46.267350 (+1056402us) ts_tablet_manager.cc:1151] Deleting tablet 07f86f24d9d14d05898e1c0f69428850
0924 19:57:46.275206 (+ 7856us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:46.335537 (+ 60331us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:46.390692 (+ 55155us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:46.422268 (+ 31576us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1055822}
I20260924 19:57:46.423744 18910 tablet_replica.cc:333] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:46.424110 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 07f86f24d9d14d05898e1c0f69428850 (table default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]) successfully deleted
I20260924 19:57:46.424453 18910 raft_consensus.cc:2243] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:46.430210 18910 raft_consensus.cc:2272] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:46.434427 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_0c6b7fd4e7bd42a3ba0d407aea438273"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:46.435995 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c6b7fd4e7bd42a3ba0d407aea438273 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:46.437673 19080 raft_consensus.cc:515] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.440115 18051 leader_election.cc:304] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:46.447319 18220 tablet_bootstrap.cc:654] T f169c47d711d4566b2004cdfd0de725d P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:46.458350 18910 ts_tablet_manager.cc:1916] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:46.460688 18780 raft_consensus.cc:493] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:46.461169 18780 raft_consensus.cc:515] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:46.466610 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431d298f513d47268284d50af2ea1436" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:46.419050 18223 consensus_queue.cc:260] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.467828 19080 leader_election.cc:290] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:46.463128 18780 leader_election.cc:290] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:46.470410 18050 leader_election.cc:343] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:46.471024 19203 raft_consensus.cc:2804] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:46.470964 17606 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_b6d3fa192eb741158adf5b69323a29de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:46.472481 17600 catalog_manager.cc:5043] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet cb111f7a189c4f00a41686d647c64e65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb111f7a189c4f00a41686d647c64e65 already in progress: creating tablet
W20260924 19:57:46.472813 17606 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6d3fa192eb741158adf5b69323a29de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:46.482968 18223 ts_tablet_manager.cc:1434] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.122s user 0.005s sys 0.000s
I20260924 19:57:46.483234 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_843b88f433f044458daa98228935a46e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:57:46.484077 18223 tablet_bootstrap.cc:492] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:46.484714 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_843b88f433f044458daa98228935a46e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:46.495637 18163 heartbeater.cc:344] Connected to a master server at 127.17.2.188:38761
I20260924 19:57:46.498052 19203 raft_consensus.cc:697] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:46.498948 19203 consensus_queue.cc:237] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:46.546540 18223 tablet_bootstrap.cc:654] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:46.563822 18910 ts_tablet_manager.cc:1929] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260924 19:57:46.564205 18910 log.cc:1199] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/201ea0020cb3430eac85e80b8c06fb07
I20260924 19:57:46.564873 18910 ts_tablet_manager.cc:1950] T 201ea0020cb3430eac85e80b8c06fb07 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:46.566798 18910 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51332 (request call id 238) took 1231 ms. Trace:
I20260924 19:57:46.572882 19051 ts_tablet_manager.cc:1929] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260924 19:57:46.581212 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_a50ccb6bd6a042bc99b503e5aa1188ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:46.582919 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a50ccb6bd6a042bc99b503e5aa1188ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:46.588032 17606 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_850e5158d78d4850b87445dac75ef7a9" } modify_external_catalogs: true
I20260924 19:57:46.567103 18910 rpcz_store.cc:276] 0924 19:57:45.335754 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.335869 (+ 115us) service_pool.cc:224] Handling call
0924 19:57:46.423602 (+1087733us) ts_tablet_manager.cc:1151] Deleting tablet 201ea0020cb3430eac85e80b8c06fb07
0924 19:57:46.423642 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:46.490380 (+ 66738us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:46.563697 (+ 73317us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:46.566738 (+ 3041us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1087148}
I20260924 19:57:46.588548 17606 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_850e5158d78d4850b87445dac75ef7a9" } modify_external_catalogs: true
I20260924 19:57:46.590984 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]) successfully deleted
W20260924 19:57:46.590826 18050 consensus_peers.cc:597] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260924 19:57:46.591948 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet d970cc92a1134a98a613d198a319cb01 because the tablet was not found. No further retry: Not found: Tablet not found: d970cc92a1134a98a613d198a319cb01
W20260924 19:57:46.592540 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet c058a49bfd5544b58b76edfd3fadbd1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c058a49bfd5544b58b76edfd3fadbd1a already in progress: creating tablet
W20260924 19:57:46.593122 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 5f116f866c2b4674b4eef531cf00af66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f116f866c2b4674b4eef531cf00af66 already in progress: creating tablet
W20260924 19:57:46.593677 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet cb111f7a189c4f00a41686d647c64e65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb111f7a189c4f00a41686d647c64e65 already in progress: creating tablet
I20260924 19:57:46.595515 19051 log.cc:1199] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/201ea0020cb3430eac85e80b8c06fb07
I20260924 19:57:46.596315 19051 ts_tablet_manager.cc:1950] T 201ea0020cb3430eac85e80b8c06fb07 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:46.616111 18223 tablet_bootstrap.cc:492] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:46.616634 18223 ts_tablet_manager.cc:1403] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.133s user 0.010s sys 0.001s
I20260924 19:57:46.619539 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:46.618960 18223 raft_consensus.cc:359] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.620572 18223 raft_consensus.cc:385] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.621425 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:46.621876 18223 raft_consensus.cc:740] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.622668 18223 consensus_queue.cc:260] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:46.624087 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 149913us
I20260924 19:57:46.624754 17753 leader_election.cc:304] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:46.625442 18780 raft_consensus.cc:2804] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:46.625830 18780 raft_consensus.cc:493] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:46.626209 18780 raft_consensus.cc:3060] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:46.639211 18223 ts_tablet_manager.cc:1434] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
W20260924 19:57:46.654659 17753 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:46.658993 17608 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_9259880c400d44d69831fe68d31b06f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:57:46.660645 17608 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9259880c400d44d69831fe68d31b06f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:57:46.660795 18220 tablet_bootstrap.cc:492] T f169c47d711d4566b2004cdfd0de725d P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:46.661224 18220 ts_tablet_manager.cc:1403] T f169c47d711d4566b2004cdfd0de725d P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.253s user 0.023s sys 0.011s
I20260924 19:57:46.662458 18780 raft_consensus.cc:515] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.663530 18220 raft_consensus.cc:359] T f169c47d711d4566b2004cdfd0de725d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:46.664191 18220 raft_consensus.cc:385] T f169c47d711d4566b2004cdfd0de725d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.664462 18220 raft_consensus.cc:740] T f169c47d711d4566b2004cdfd0de725d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.665127 18220 consensus_queue.cc:260] T f169c47d711d4566b2004cdfd0de725d P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:46.671531 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1137318a9fa4a0781f718dc34c4ecbe" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:46.674027 18220 ts_tablet_manager.cc:1434] T f169c47d711d4566b2004cdfd0de725d P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260924 19:57:46.675009 18220 tablet_bootstrap.cc:492] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:46.678365 19135 raft_consensus.cc:493] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:46.678817 19135 raft_consensus.cc:515] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.680747 19135 leader_election.cc:290] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:46.684420 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1137318a9fa4a0781f718dc34c4ecbe" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:46.689185 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df3dcf4ddcf4efa9aa047da56dfb7a7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:46.697698 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 234) took 1217 ms. Trace:
I20260924 19:57:46.698061 19051 rpcz_store.cc:276] 0924 19:57:45.480272 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.482505 (+ 2233us) service_pool.cc:224] Handling call
0924 19:57:46.299712 (+817207us) ts_tablet_manager.cc:1151] Deleting tablet 201ea0020cb3430eac85e80b8c06fb07
0924 19:57:46.299757 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:46.505409 (+205652us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:46.572750 (+ 67341us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:46.697639 (+124889us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":808786}
I20260924 19:57:46.699834 18223 tablet_bootstrap.cc:492] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:46.705319 17609 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]). Expected version 1 got 0
I20260924 19:57:46.707016 19051 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 236) took 1215 ms. Trace:
I20260924 19:57:46.707455 19051 rpcz_store.cc:276] 0924 19:57:45.491757 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.491854 (+ 97us) service_pool.cc:224] Handling call
0924 19:57:46.699005 (+1207151us) ts_tablet_manager.cc:1151] Deleting tablet ba46ac86b242407abd3e8dbf09f12efa
0924 19:57:46.699048 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:46.706946 (+ 7898us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206633}
I20260924 19:57:46.713156 18223 tablet_bootstrap.cc:654] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:46.713799 18220 tablet_bootstrap.cc:654] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:46.715862 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 201ea0020cb3430eac85e80b8c06fb07 (table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]) successfully deleted
W20260924 19:57:46.716670 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet ba46ac86b242407abd3e8dbf09f12efa because the tablet was not found. No further retry: Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
W20260924 19:57:46.717365 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet c058a49bfd5544b58b76edfd3fadbd1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c058a49bfd5544b58b76edfd3fadbd1a already in progress: creating tablet
W20260924 19:57:46.718001 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 5f116f866c2b4674b4eef531cf00af66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f116f866c2b4674b4eef531cf00af66 already in progress: creating tablet
W20260924 19:57:46.719696 17599 catalog_manager.cc:5043] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet cb111f7a189c4f00a41686d647c64e65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb111f7a189c4f00a41686d647c64e65 already in progress: creating tablet
I20260924 19:57:46.736601 18220 tablet_bootstrap.cc:492] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:46.737026 18220 ts_tablet_manager.cc:1403] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.003s
I20260924 19:57:46.741317 18223 tablet_bootstrap.cc:492] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:46.746582 18223 ts_tablet_manager.cc:1403] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.000s
I20260924 19:57:46.665625 18224 tablet_bootstrap.cc:492] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:46.749080 18223 raft_consensus.cc:359] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:46.749722 18223 raft_consensus.cc:385] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.749981 18223 raft_consensus.cc:740] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.750438 18224 ts_tablet_manager.cc:1403] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.403s user 0.009s sys 0.003s
I20260924 19:57:46.751372 17812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5df3dcf4ddcf4efa9aa047da56dfb7a7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:46.753094 17906 leader_election.cc:304] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:46.754120 19135 raft_consensus.cc:2749] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:46.756657 19213 raft_consensus.cc:493] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:46.757112 19213 raft_consensus.cc:515] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.758972 19213 leader_election.cc:290] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:46.761672 17812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431d298f513d47268284d50af2ea1436" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:46.762192 17812 raft_consensus.cc:3060] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:46.750645 18223 consensus_queue.cc:260] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:46.753317 18224 raft_consensus.cc:359] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.767387 18224 raft_consensus.cc:385] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.767657 18224 raft_consensus.cc:740] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.767815 18223 ts_tablet_manager.cc:1434] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260924 19:57:46.767781 18220 raft_consensus.cc:359] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.768435 18220 raft_consensus.cc:385] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.768230 18224 consensus_queue.cc:260] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.768657 18223 tablet_bootstrap.cc:492] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:46.768674 18220 raft_consensus.cc:740] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.769269 18220 consensus_queue.cc:260] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.780318 18780 leader_election.cc:290] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:46.790259 18223 tablet_bootstrap.cc:654] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:46.798328 17608 master_service.cc:960] ReplaceTablet: received request to replace tablet 8aef0359a77240679240823612507ee8 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:46.827203 17823 raft_consensus.cc:1275] T 274f3decdc7e42c0917bf28151c4da71 P c4f6e3a7b06148918d230c77ab0c8202 [term 2 FOLLOWER]: Refusing update from remote peer f74e6ed351284d7fb110e44336ce643a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 19:57:46.828794 19170 consensus_queue.cc:1048] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:46.834159 18220 ts_tablet_manager.cc:1434] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.097s user 0.008s sys 0.000s
I20260924 19:57:46.841724 19242 tablet.cc:1751] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:46.847020 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9e7c5b28fea4dbda802702c8b85ec54" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:46.849797 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9e7c5b28fea4dbda802702c8b85ec54" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:46.835103 18220 tablet_bootstrap.cc:492] T f43f2a255c644ca6a68a8d867798fc1a P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:46.858959 18223 tablet_bootstrap.cc:492] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:46.859448 18223 ts_tablet_manager.cc:1403] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.091s user 0.002s sys 0.009s
I20260924 19:57:46.859364 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aef0359a77240679240823612507ee8" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:46.861750 18223 raft_consensus.cc:359] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.866289 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aef0359a77240679240823612507ee8" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:46.866829 18111 raft_consensus.cc:3060] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:46.867225 18223 raft_consensus.cc:385] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.867522 18223 raft_consensus.cc:740] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.873404 18111 raft_consensus.cc:2468] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:46.875402 18223 consensus_queue.cc:260] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.877270 18223 ts_tablet_manager.cc:1434] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.018s user 0.001s sys 0.003s
I20260924 19:57:46.878072 18223 tablet_bootstrap.cc:492] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:46.880223 17753 leader_election.cc:343] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:46.890143 18224 ts_tablet_manager.cc:1434] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.139s user 0.011s sys 0.001s
I20260924 19:57:46.891798 18224 tablet_bootstrap.cc:492] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:46.893007 17812 raft_consensus.cc:2468] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:46.898401 18224 tablet_bootstrap.cc:654] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:46.905355 18223 tablet_bootstrap.cc:654] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:46.867131 18220 tablet_bootstrap.cc:654] T f43f2a255c644ca6a68a8d867798fc1a P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:46.909934 17753 leader_election.cc:343] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:46.911728 17753 leader_election.cc:304] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:46.917129 19243 tablet.cc:1751] T 6bd442f16492495f84bb8363b9bade97 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260924 19:57:46.930701 19232 log.cc:927] Time spent T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: Append to log took a long time: real 0.213s user 0.017s sys 0.004s
I20260924 19:57:46.931906 19213 raft_consensus.cc:2804] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:46.932358 19213 raft_consensus.cc:697] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:46.933179 19213 consensus_queue.cc:237] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:46.945748 19176 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:46.946362 19176 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:46.953780 19151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ff67c5d57d474a9d92cc4c6de1287e. Latest consensus state: current_term: 3 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:46.954507 19151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:46.958048 18224 tablet_bootstrap.cc:492] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:46.958448 18224 ts_tablet_manager.cc:1403] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.000s
I20260924 19:57:46.960955 18224 raft_consensus.cc:359] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:46.961561 18224 raft_consensus.cc:385] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:46.961804 18224 raft_consensus.cc:740] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:46.966310 18224 consensus_queue.cc:260] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:46.964382 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 52733.5us
W20260924 19:57:46.967824 17753 leader_election.cc:343] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:46.968259 17753 leader_election.cc:304] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:46.968951 19213 raft_consensus.cc:2749] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:46.969324 17753 leader_election.cc:343] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:46.969731 17753 leader_election.cc:304] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:46.970238 19250 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:57:46.970388 19213 raft_consensus.cc:2749] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:46.970808 17753 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:46.977207 18220 tablet_bootstrap.cc:492] T f43f2a255c644ca6a68a8d867798fc1a P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:46.977708 18220 ts_tablet_manager.cc:1403] T f43f2a255c644ca6a68a8d867798fc1a P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.143s user 0.010s sys 0.000s
I20260924 19:57:46.978436 18224 ts_tablet_manager.cc:1434] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260924 19:57:46.986021 17795 tablet_service.cc:1511] Processing CreateTablet for tablet 3d25793b65da4315b6f95726374ea46c (DEFAULT_TABLE table=default.Table_5cc282e71cd74150a849aff1620d54f1 [id=010f7a5a46034f008e83a6492524b778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:46.986191 17945 tablet_service.cc:1511] Processing CreateTablet for tablet 3d25793b65da4315b6f95726374ea46c (DEFAULT_TABLE table=default.Table_5cc282e71cd74150a849aff1620d54f1 [id=010f7a5a46034f008e83a6492524b778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:46.987131 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d25793b65da4315b6f95726374ea46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:46.987721 19176 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:46.989338 17800 tablet_service.cc:1511] Processing CreateTablet for tablet a00e3626ea154dc5a959f0fbb4e849dc (DEFAULT_TABLE table=default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:46.992797 17940 tablet_service.cc:1511] Processing CreateTablet for tablet a00e3626ea154dc5a959f0fbb4e849dc (DEFAULT_TABLE table=default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:46.993842 17940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a00e3626ea154dc5a959f0fbb4e849dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:46.986613 18220 raft_consensus.cc:359] T f43f2a255c644ca6a68a8d867798fc1a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.002516 18220 raft_consensus.cc:385] T f43f2a255c644ca6a68a8d867798fc1a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.002918 18220 raft_consensus.cc:740] T f43f2a255c644ca6a68a8d867798fc1a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.003608 18220 consensus_queue.cc:260] T f43f2a255c644ca6a68a8d867798fc1a P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.005784 18220 ts_tablet_manager.cc:1434] T f43f2a255c644ca6a68a8d867798fc1a P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260924 19:57:47.038756 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 249 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:47.006981 18220 tablet_bootstrap.cc:492] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:47.054039 19213 raft_consensus.cc:493] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.054540 19213 raft_consensus.cc:515] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.057518 18091 tablet_service.cc:1511] Processing CreateTablet for tablet a00e3626ea154dc5a959f0fbb4e849dc (DEFAULT_TABLE table=default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:47.056494 19213 leader_election.cc:290] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:47.058673 18091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a00e3626ea154dc5a959f0fbb4e849dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:47.061378 18113 raft_consensus.cc:1275] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 2 FOLLOWER]: Refusing update from remote peer f74e6ed351284d7fb110e44336ce643a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260924 19:57:47.061307 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 3d25793b65da4315b6f95726374ea46c (DEFAULT_TABLE table=default.Table_5cc282e71cd74150a849aff1620d54f1 [id=010f7a5a46034f008e83a6492524b778]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:57:47.062269 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d25793b65da4315b6f95726374ea46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:47.064570 19135 consensus_queue.cc:1048] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260924 19:57:47.066179 19166 raft_consensus.cc:493] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.066612 19166 raft_consensus.cc:515] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.068423 19166 leader_election.cc:290] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.075452 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924bc897f48b44069f1fb2ee71e26db7" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:47.080320 18939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55534 (request call id 0) took 5530 ms. Trace:
I20260924 19:57:47.082616 19203 raft_consensus.cc:493] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.078401 18220 tablet_bootstrap.cc:654] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:47.119027 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 44030.8us
I20260924 19:57:47.129663 17608 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 8aef0359a77240679240823612507ee8
I20260924 19:57:47.078795 19135 raft_consensus.cc:493] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.130869 19135 raft_consensus.cc:515] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.132864 17796 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aef0359a77240679240823612507ee8 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b0b011357674a42a09d83f9384abaf1) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:47.134016 19135 leader_election.cc:290] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.134699 19135 raft_consensus.cc:493] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.135093 19135 raft_consensus.cc:515] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.137583 17608 catalog_manager.cc:6934] ReplaceTablet: tablet 8aef0359a77240679240823612507ee8 deleted and replaced by tablet 2b0b011357674a42a09d83f9384abaf1
I20260924 19:57:47.137706 19135 leader_election.cc:290] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.083014 19203 raft_consensus.cc:515] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.151029 19203 leader_election.cc:290] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:47.165177 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 45514.5us
I20260924 19:57:47.167313 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bafaab10ce564b1b97659eccc0ab0967" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:47.168115 17967 raft_consensus.cc:3060] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.192950 18223 tablet_bootstrap.cc:492] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:47.193454 18223 ts_tablet_manager.cc:1403] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.316s user 0.011s sys 0.002s
I20260924 19:57:47.196107 18223 raft_consensus.cc:359] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.196798 18223 raft_consensus.cc:385] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.197108 18223 raft_consensus.cc:740] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.197772 18223 consensus_queue.cc:260] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.215459 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df722b0bbfd04ce2943f0e5b8058c50f" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:47.215771 19262 raft_consensus.cc:493] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.218466 19262 raft_consensus.cc:515] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:47.220387 18223 ts_tablet_manager.cc:1434] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20260924 19:57:47.220789 19262 leader_election.cc:290] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:47.223111 18223 tablet_bootstrap.cc:492] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:47.223692 17941 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aef0359a77240679240823612507ee8 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b0b011357674a42a09d83f9384abaf1) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:47.233049 17906 leader_election.cc:343] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 44199d2655f14dcf97afe2180789aae4
W20260924 19:57:47.239413 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 8aef0359a77240679240823612507ee8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aef0359a77240679240823612507ee8 already in progress: creating tablet
I20260924 19:57:47.242579 18220 tablet_bootstrap.cc:492] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:47.243010 18220 ts_tablet_manager.cc:1403] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.236s user 0.012s sys 0.008s
I20260924 19:57:47.245271 18220 raft_consensus.cc:359] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.245880 18220 raft_consensus.cc:385] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.246133 18220 raft_consensus.cc:740] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.246774 18220 consensus_queue.cc:260] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.248955 18220 ts_tablet_manager.cc:1434] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 19:57:47.249825 18220 tablet_bootstrap.cc:492] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:47.253566 18928 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" instance_seqno: 1790279840847753) as {username='slave'} at 127.17.2.130:55157; Asking this server to re-register.
I20260924 19:57:47.257061 18019 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:47.257762 18220 tablet_bootstrap.cc:654] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:47.262521 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df722b0bbfd04ce2943f0e5b8058c50f" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:47.263610 17815 raft_consensus.cc:2393] T df722b0bbfd04ce2943f0e5b8058c50f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:47.272946 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aef0359a77240679240823612507ee8 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b0b011357674a42a09d83f9384abaf1) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:47.273232 17905 leader_election.cc:304] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
W20260924 19:57:47.273703 18050 leader_election.cc:343] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:47.274421 17905 leader_election.cc:343] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 44199d2655f14dcf97afe2180789aae4
I20260924 19:57:47.274482 19170 raft_consensus.cc:3060] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.274817 17905 leader_election.cc:304] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:47.277258 19260 raft_consensus.cc:2749] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:47.281996 18928 ts_manager.cc:194] Registered new tserver with Master: f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:47.162904 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:47.286303 19170 raft_consensus.cc:2749] T df722b0bbfd04ce2943f0e5b8058c50f P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:47.289920 18925 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" instance_seqno: 1790279843119359) as {username='slave'} at 127.17.2.131:43679; Asking this server to re-register.
I20260924 19:57:47.291961 17795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d25793b65da4315b6f95726374ea46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:47.292548 17800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a00e3626ea154dc5a959f0fbb4e849dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:57:47.296025 18224 tablet_bootstrap.cc:492] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:47.296005 19272 tablet_replica.cc:333] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:47.297014 19272 raft_consensus.cc:2243] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:47.305260 18224 tablet_bootstrap.cc:654] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:47.312036 19264 tablet_replica.cc:333] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:47.321480 18220 tablet_bootstrap.cc:492] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:47.321930 18220 ts_tablet_manager.cc:1403] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.072s user 0.023s sys 0.009s
I20260924 19:57:47.324350 18220 raft_consensus.cc:359] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.324851 19264 raft_consensus.cc:2243] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:47.325038 18220 raft_consensus.cc:385] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.325117 18224 tablet_bootstrap.cc:492] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:47.325311 18220 raft_consensus.cc:740] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.325500 18224 ts_tablet_manager.cc:1403] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.000s
I20260924 19:57:47.326082 19264 pending_rounds.cc:70] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: Trying to abort 1 pending ops.
I20260924 19:57:47.326411 19264 pending_rounds.cc:77] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986334957387776 op_type: NO_OP noop_request { }
I20260924 19:57:47.326874 19264 raft_consensus.cc:2889] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:57:47.327333 19264 raft_consensus.cc:2272] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:47.330312 18224 raft_consensus.cc:359] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.333978 19272 raft_consensus.cc:2272] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:47.335865 19272 ts_tablet_manager.cc:1916] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:47.337175 18166 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:47.342342 18220 consensus_queue.cc:260] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.350903 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:47.352208 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 2e6d926ff7c24901a15b0c7b2239fbad (table default.Table_7c27cdce0a3b40f69142de52aaf39493 [id=65f397bcbc164d0caa6cecae1bab74da]). Expected version 1 got 0
I20260924 19:57:47.354209 18220 ts_tablet_manager.cc:1434] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260924 19:57:47.354898 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet f43f2a255c644ca6a68a8d867798fc1a (table default.Table_4e3a3158cde24302806b7ba7827831c4 [id=fa0b10df171340868b0305a28d192ea5]). Expected version 1 got 0
I20260924 19:57:47.356937 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]). Expected version 1 got 0
I20260924 19:57:47.357885 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_88352246addd43f4be871c983413b2de [id=fa469a44fb3741d39036337129cc36a1]). Expected version 2 got 1
I20260924 19:57:47.358389 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 3 got 0
W20260924 19:57:47.261675 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 290859us
I20260924 19:57:47.359074 18928 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" instance_seqno: 1790279838438873) as {username='slave'} at 127.17.2.129:59015; Asking this server to re-register.
I20260924 19:57:47.361372 18051 leader_election.cc:304] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:47.363327 19262 raft_consensus.cc:2804] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:47.363746 19262 raft_consensus.cc:697] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:47.364485 19262 consensus_queue.cc:237] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.372731 18223 tablet_bootstrap.cc:700] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent opening tablet: real 0.119s user 0.002s sys 0.001s
I20260924 19:57:47.373349 18223 tablet_bootstrap.cc:654] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:47.330965 18224 raft_consensus.cc:385] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.375339 18224 raft_consensus.cc:740] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.376133 18224 consensus_queue.cc:260] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:47.332532 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 58299.3us
I20260924 19:57:47.378413 18224 ts_tablet_manager.cc:1434] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.053s user 0.005s sys 0.000s
W20260924 19:57:47.378765 18050 leader_election.cc:343] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:47.383287 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78801b8d0b664217a74b20160ba2d8ac" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:47.388504 19260 raft_consensus.cc:493] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.388933 19260 raft_consensus.cc:515] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.389956 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 4 candidate_status { last_received { term: 3 index: 249 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:47.392086 18903 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:47.390997 19260 leader_election.cc:290] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.393268 19176 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 19:57:47.394029 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924bc897f48b44069f1fb2ee71e26db7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:47.394273 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924bc897f48b44069f1fb2ee71e26db7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:47.394615 17823 raft_consensus.cc:2468] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:47.394899 18113 raft_consensus.cc:2468] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:47.396136 18220 tablet_bootstrap.cc:492] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:47.396386 17906 leader_election.cc:304] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:47.405350 19272 ts_tablet_manager.cc:1929] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:47.408679 19264 ts_tablet_manager.cc:1916] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:47.410216 19272 log.cc:1199] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/8aef0359a77240679240823612507ee8
I20260924 19:57:47.411692 19272 ts_tablet_manager.cc:1950] T 8aef0359a77240679240823612507ee8 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:47.420195 19260 raft_consensus.cc:2804] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:47.420571 19260 raft_consensus.cc:493] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:47.420887 19260 raft_consensus.cc:3060] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.425295 18220 tablet_bootstrap.cc:654] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:47.426800 17972 raft_consensus.cc:3060] T f169c47d711d4566b2004cdfd0de725d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.431371 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57008266b80345e0a3f8e60bd7e2d62b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:47.436374 19263 raft_consensus.cc:493] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.436821 19263 raft_consensus.cc:515] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.438763 19263 leader_election.cc:290] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.417877 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924bc897f48b44069f1fb2ee71e26db7" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:47.440184 17823 raft_consensus.cc:2468] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:47.442008 18928 ts_manager.cc:194] Registered new tserver with Master: eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:47.438526 19260 raft_consensus.cc:515] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.443537 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78801b8d0b664217a74b20160ba2d8ac" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:47.448426 18050 consensus_peers.cc:597] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:47.448374 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924bc897f48b44069f1fb2ee71e26db7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:47.448855 17823 raft_consensus.cc:3060] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:47.448904 18050 leader_election.cc:343] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:47.450699 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3192f66a492442c1bf4137cfe7e13bc9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:47.451009 17972 raft_consensus.cc:3060] T f43f2a255c644ca6a68a8d867798fc1a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.455631 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 8aef0359a77240679240823612507ee8 (table default.Table_5aeda90fd8e3425a8c49a239533690c9 [id=63622937f5a64a05b56aa1310407ec99]) successfully deleted
I20260924 19:57:47.456466 17823 raft_consensus.cc:2468] T 924bc897f48b44069f1fb2ee71e26db7 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:47.457683 17906 leader_election.cc:304] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:47.458410 19170 raft_consensus.cc:2804] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:47.458822 19170 raft_consensus.cc:697] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:47.459560 17872 heartbeater.cc:461] Registering TS with master...
I20260924 19:57:47.459586 19170 consensus_queue.cc:237] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.464339 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "924bc897f48b44069f1fb2ee71e26db7" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:47.464941 18111 raft_consensus.cc:3060] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.465312 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bafaab10ce564b1b97659eccc0ab0967" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:47.465837 17821 raft_consensus.cc:2468] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:47.468310 18223 tablet_bootstrap.cc:492] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:47.468737 18223 ts_tablet_manager.cc:1403] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.246s user 0.005s sys 0.004s
I20260924 19:57:47.470450 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57008266b80345e0a3f8e60bd7e2d62b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:47.471029 17821 raft_consensus.cc:2468] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:47.472121 18111 raft_consensus.cc:2468] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:47.472180 19260 leader_election.cc:290] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.482439 18220 tablet_bootstrap.cc:492] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:47.482856 18220 ts_tablet_manager.cc:1403] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.087s user 0.010s sys 0.001s
I20260924 19:57:47.483383 18224 tablet_bootstrap.cc:492] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:47.485445 18220 raft_consensus.cc:359] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.486074 18220 raft_consensus.cc:385] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.486328 18220 raft_consensus.cc:740] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.495369 19263 consensus_queue.cc:1048] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:47.497107 18223 raft_consensus.cc:359] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:47.497718 18223 raft_consensus.cc:385] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.497972 18223 raft_consensus.cc:740] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.498565 18223 consensus_queue.cc:260] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:47.504820 18223 ts_tablet_manager.cc:1434] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20260924 19:57:47.505771 18223 tablet_bootstrap.cc:492] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:47.506145 19262 consensus_queue.cc:1048] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:47.508344 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 58927.7us
I20260924 19:57:47.509663 18220 consensus_queue.cc:260] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.511288 18051 leader_election.cc:304] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:47.511762 18220 ts_tablet_manager.cc:1434] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260924 19:57:47.512115 19263 raft_consensus.cc:2764] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 19:57:47.515730 18051 leader_election.cc:304] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:47.516422 19263 raft_consensus.cc:2804] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260924 19:57:47.515692 18050 leader_election.cc:343] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:47.516794 19263 raft_consensus.cc:493] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:47.517220 19263 raft_consensus.cc:3060] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.520151 19135 raft_consensus.cc:493] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.520565 19135 raft_consensus.cc:515] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.522490 19135 leader_election.cc:290] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.358989 17611 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet d129e0979a584f40ba9a975349cf15a5 (table default.Table_a12f8684de2041ec83f2fee77829afdd [id=94cea8de30ca4010a95f142ac8408653]). Expected version 1 got 0
I20260924 19:57:47.524360 17812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8aef0359a77240679240823612507ee8" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:47.525508 17615 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 1d95d6e48173436c8c59130464413ea1
I20260924 19:57:47.528998 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d95d6e48173436c8c59130464413ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:46 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:47.526988 17615 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 259) took 1477 ms. Trace:
I20260924 19:57:47.529331 17615 rpcz_store.cc:276] 0924 19:57:46.049300 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:46.049417 (+ 117us) service_pool.cc:224] Handling call
0924 19:57:46.077433 (+ 28016us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:46.078031 (+ 598us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:46.078163 (+ 132us) catalog_manager.cc:2863] Locking tablets
0924 19:57:46.079670 (+ 1507us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:46.094194 (+ 14524us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:46.094443 (+ 249us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:47.525396 (+1430953us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:47.526924 (+ 1528us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:46.081619 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:46.082069 (+ 450us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:46.082098 (+ 29us) write_op.cc:435] Acquired schema lock
0924 19:57:46.082116 (+ 18us) tablet.cc:662] Decoding operations
0924 19:57:46.082443 (+ 327us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:46.082488 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:46.082513 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:46.082820 (+ 307us) tablet.cc:701] Row locks acquired
0924 19:57:46.082838 (+ 18us) write_op.cc:260] PREPARE: finished
0924 19:57:46.082979 (+ 141us) write_op.cc:270] Start()
0924 19:57:46.083073 (+ 94us) write_op.cc:276] Timestamp: P: 1790279866082914 usec, L: 0
0924 19:57:46.083089 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:57:46.083648 (+ 559us) log.cc:844] Serialized 728 byte log entry
0924 19:57:46.091460 (+ 7812us) op_driver.cc:464] REPLICATION: finished
0924 19:57:46.091774 (+ 314us) write_op.cc:301] APPLY: starting
0924 19:57:46.091834 (+ 60us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:46.091959 (+ 125us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:46.092060 (+ 101us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:46.092324 (+ 264us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:46.092482 (+ 158us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:46.092528 (+ 46us) write_op.cc:312] APPLY: finished
0924 19:57:46.092723 (+ 195us) log.cc:844] Serialized 29 byte log entry
0924 19:57:46.092894 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:46.092975 (+ 81us) write_op.cc:454] Released schema lock
0924 19:57:46.093018 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:46.093300 (+ 282us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:46.093335 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":168,"apply.run_cpu_time_us":1478,"apply.run_wall_time_us":1478,"num_ops":2,"prepare.queue_time_us":123,"prepare.run_cpu_time_us":2329,"prepare.run_wall_time_us":2329,"raft.queue_time_us":1059,"raft.run_cpu_time_us":766,"raft.run_wall_time_us":767,"replication_time_us":8269}]]}
I20260924 19:57:47.532683 18220 tablet_bootstrap.cc:492] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
W20260924 19:57:47.532998 17906 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Illegal state: Tablet not RUNNING: STOPPED
I20260924 19:57:47.534332 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d95d6e48173436c8c59130464413ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:46 UTC) from {username='slave'} at 127.0.0.1:51332
W20260924 19:57:47.535825 17599 catalog_manager.cc:5043] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 1d95d6e48173436c8c59130464413ea1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d95d6e48173436c8c59130464413ea1 already in progress: creating tablet
I20260924 19:57:47.540225 19259 raft_consensus.cc:493] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.540680 19259 raft_consensus.cc:515] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.542584 19259 leader_election.cc:290] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.551595 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d95d6e48173436c8c59130464413ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:46 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:47.552404 19272 tablet_replica.cc:333] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:47.553336 19272 raft_consensus.cc:2243] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:47.553535 18220 tablet_bootstrap.cc:654] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:47.553823 19272 raft_consensus.cc:2272] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:47.556281 19272 ts_tablet_manager.cc:1916] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:47.576249 18224 tablet_bootstrap.cc:654] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:47.577243 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca37ddd22f5d4bcab2862553408447ce" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:47.578894 17905 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 8aef0359a77240679240823612507ee8
I20260924 19:57:47.582708 17905 leader_election.cc:304] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:47.583968 19278 tablet.cc:1751] T 6bd442f16492495f84bb8363b9bade97 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:47.581955 19260 raft_consensus.cc:493] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.588466 19260 raft_consensus.cc:515] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:47.589010 18050 leader_election.cc:343] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:47.591497 19135 raft_consensus.cc:2749] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:47.593725 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "484517ee7df8463bb70d201ead1c6835" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:47.594419 17821 raft_consensus.cc:2393] T 484517ee7df8463bb70d201ead1c6835 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:47.594707 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "484517ee7df8463bb70d201ead1c6835" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:47.597602 19213 raft_consensus.cc:493] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.598071 19213 raft_consensus.cc:515] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.599941 19213 leader_election.cc:290] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:47.604565 19263 raft_consensus.cc:515] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.606680 19260 leader_election.cc:290] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.608379 18925 ts_manager.cc:194] Registered new tserver with Master: c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.609678 18051 leader_election.cc:304] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:47.610786 19259 raft_consensus.cc:2804] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:47.611122 19259 raft_consensus.cc:493] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:47.611445 19259 raft_consensus.cc:3060] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.611938 18220 tablet_bootstrap.cc:492] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:47.612370 18220 ts_tablet_manager.cc:1403] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.000s
I20260924 19:57:47.615641 18220 raft_consensus.cc:359] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.616281 18220 raft_consensus.cc:385] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.616545 18220 raft_consensus.cc:740] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.617161 18220 consensus_queue.cc:260] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.620219 18223 tablet_bootstrap.cc:654] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:47.621029 19259 raft_consensus.cc:515] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:47.623569 19259 leader_election.cc:290] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
W20260924 19:57:47.645995 17599 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet d129e0979a584f40ba9a975349cf15a5 (table default.Table_a12f8684de2041ec83f2fee77829afdd [id=94cea8de30ca4010a95f142ac8408653], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d129e0979a584f40ba9a975349cf15a5 consensus state has no leader
I20260924 19:57:47.648772 19284 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:47.654639 19284 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:47.621395 19135 raft_consensus.cc:493] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.655697 19135 raft_consensus.cc:515] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.657692 19135 leader_election.cc:290] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.631713 17966 raft_consensus.cc:3060] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.632725 17905 leader_election.cc:304] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:47.664747 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bafaab10ce564b1b97659eccc0ab0967" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:47.665956 19135 raft_consensus.cc:2749] T 484517ee7df8463bb70d201ead1c6835 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:47.619130 18220 ts_tablet_manager.cc:1434] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260924 19:57:47.667608 18220 tablet_bootstrap.cc:492] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:47.669322 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b99ff1920274831b58d8283bd592818" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:47.670186 17821 raft_consensus.cc:2393] T 0b99ff1920274831b58d8283bd592818 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:47.677654 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57008266b80345e0a3f8e60bd7e2d62b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
W20260924 19:57:47.681931 17753 leader_election.cc:343] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:47.636883 19262 consensus_queue.cc:1048] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:47.652971 19204 consensus_queue.cc:1048] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:47.682335 18938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 321 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:47.682911 18938 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f88e00b235f48bb82e6a3c9c9f51fa6 in term 3.
W20260924 19:57:47.672004 18051 leader_election.cc:343] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 3192f66a492442c1bf4137cfe7e13bc9
I20260924 19:57:47.684233 17522 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d, 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters:
I20260924 19:57:47.684690 18051 leader_election.cc:304] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:47.689272 19262 raft_consensus.cc:2749] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:47.693295 19286 tablet.cc:1751] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:47.695734 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 321 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:47.707666 19284 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:47.710704 19284 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260924 19:57:47.712132 19284 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:47.712972 19284 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 3 FOLLOWER]: Advancing to term 4
W20260924 19:57:47.716594 17753 leader_election.cc:343] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:47.716960 17753 leader_election.cc:304] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:47.696501 18223 tablet_bootstrap.cc:492] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:47.692693 18224 tablet_bootstrap.cc:492] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:47.731386 18223 ts_tablet_manager.cc:1403] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.226s user 0.004s sys 0.006s
I20260924 19:57:47.731480 18224 ts_tablet_manager.cc:1403] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.352s user 0.011s sys 0.000s
I20260924 19:57:47.733913 18223 raft_consensus.cc:359] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:47.735051 18223 raft_consensus.cc:385] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.735145 19213 raft_consensus.cc:2749] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:47.735357 18223 raft_consensus.cc:740] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.736482 18223 consensus_queue.cc:260] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:47.736742 18224 raft_consensus.cc:359] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.737434 18224 raft_consensus.cc:385] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.737726 18224 raft_consensus.cc:740] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.738406 18223 ts_tablet_manager.cc:1434] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 19:57:47.738448 18224 consensus_queue.cc:260] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.739256 18223 tablet_bootstrap.cc:492] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:47.740090 18224 ts_tablet_manager.cc:1434] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260924 19:57:47.740942 18224 tablet_bootstrap.cc:492] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:47.696987 19264 ts_tablet_manager.cc:1929] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:47.747381 19264 log.cc:1199] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/8aef0359a77240679240823612507ee8
I20260924 19:57:47.748427 19264 ts_tablet_manager.cc:1950] T 8aef0359a77240679240823612507ee8 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:47.680553 19263 leader_election.cc:290] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.718605 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b99ff1920274831b58d8283bd592818" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:47.719837 18050 leader_election.cc:343] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:47.758081 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d930e7f4a6b7471899dab1b28e40f93d" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:47.758738 18220 tablet_bootstrap.cc:654] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:47.760934 18223 tablet_bootstrap.cc:654] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:47.761390 19213 raft_consensus.cc:493] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.761894 19213 raft_consensus.cc:515] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:57:47.763732 18050 leader_election.cc:343] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:47.763790 19213 leader_election.cc:290] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:47.765553 19263 consensus_queue.cc:1048] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:47.767755 19213 raft_consensus.cc:493] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.768203 19213 raft_consensus.cc:515] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.770036 19213 leader_election.cc:290] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:47.771349 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 8aef0359a77240679240823612507ee8 (table default.Table_5aeda90fd8e3425a8c49a239533690c9 [id=63622937f5a64a05b56aa1310407ec99]) successfully deleted
I20260924 19:57:47.772307 19264 tablet_replica.cc:333] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:57:47.772182 19243 tablet.cc:1751] T f43f2a255c644ca6a68a8d867798fc1a P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:47.774166 17905 leader_election.cc:304] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:47.774941 19135 raft_consensus.cc:3060] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.776324 19284 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:47.779039 18938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 321 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d"
I20260924 19:57:47.779633 18938 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 3 FOLLOWER]: Advancing to term 4
I20260924 19:57:47.781641 19135 raft_consensus.cc:2749] T 0b99ff1920274831b58d8283bd592818 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:47.785794 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d930e7f4a6b7471899dab1b28e40f93d" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:47.788117 19272 ts_tablet_manager.cc:1929] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:47.788568 19272 log.cc:1199] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/1d95d6e48173436c8c59130464413ea1
I20260924 19:57:47.789310 19272 ts_tablet_manager.cc:1950] T 1d95d6e48173436c8c59130464413ea1 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:47.789788 18938 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f88e00b235f48bb82e6a3c9c9f51fa6 in term 4.
I20260924 19:57:47.791143 17522 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d, 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters:
I20260924 19:57:47.792086 19287 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 4 FOLLOWER]: Leader election won for term 4
I20260924 19:57:47.773293 19264 raft_consensus.cc:2243] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:47.807727 19264 raft_consensus.cc:2272] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:47.809782 19264 ts_tablet_manager.cc:1916] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:47.810298 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 4 candidate_status { last_received { term: 3 index: 321 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e"
I20260924 19:57:47.813419 19284 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 4 election: Requested vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:47.817086 18223 tablet_bootstrap.cc:492] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:47.817584 18223 ts_tablet_manager.cc:1403] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.000s
I20260924 19:57:47.819638 19287 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 4 LEADER]: Becoming Leader. State: Replica: 6f88e00b235f48bb82e6a3c9c9f51fa6, State: Running, Role: LEADER
I20260924 19:57:47.820683 19287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 3.321, Last appended by leader: 321, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:47.826378 18223 raft_consensus.cc:359] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.828759 19284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f88e00b235f48bb82e6a3c9c9f51fa6. Latest consensus state: current_term: 4 leader_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:47.836680 19284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:47.827014 18223 raft_consensus.cc:385] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.847478 18223 raft_consensus.cc:740] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.848474 18223 consensus_queue.cc:260] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.859666 18220 tablet_bootstrap.cc:492] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:47.860054 18220 ts_tablet_manager.cc:1403] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.193s user 0.011s sys 0.001s
I20260924 19:57:47.862417 18220 raft_consensus.cc:359] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.867645 18223 ts_tablet_manager.cc:1434] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.050s user 0.008s sys 0.000s
I20260924 19:57:47.863061 18220 raft_consensus.cc:385] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:47.876529 18220 raft_consensus.cc:740] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:47.877167 18220 consensus_queue.cc:260] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.879088 18220 ts_tablet_manager.cc:1434] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20260924 19:57:47.886353 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 1d95d6e48173436c8c59130464413ea1 (table default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]) successfully deleted
I20260924 19:57:47.904808 18223 tablet_bootstrap.cc:492] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:47.914783 19135 raft_consensus.cc:493] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.915256 19135 raft_consensus.cc:515] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.918023 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3192f66a492442c1bf4137cfe7e13bc9" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:47.918646 18113 raft_consensus.cc:2468] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
W20260924 19:57:47.919027 17906 leader_election.cc:343] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 3192f66a492442c1bf4137cfe7e13bc9
I20260924 19:57:47.917671 19135 leader_election.cc:290] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.930622 18220 tablet_bootstrap.cc:492] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:47.885136 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57008266b80345e0a3f8e60bd7e2d62b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:47.933691 17815 raft_consensus.cc:3060] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.934814 17905 leader_election.cc:304] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
I20260924 19:57:47.935177 19204 raft_consensus.cc:493] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.935777 19204 raft_consensus.cc:515] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:47.937511 18220 tablet_bootstrap.cc:654] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:47.938620 19204 leader_election.cc:290] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:47.939090 19135 raft_consensus.cc:2804] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:47.939524 19135 raft_consensus.cc:493] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:47.939906 19135 raft_consensus.cc:3060] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.946698 19135 raft_consensus.cc:515] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.948725 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bafaab10ce564b1b97659eccc0ab0967" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:47.949283 17817 raft_consensus.cc:3060] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.951310 19264 ts_tablet_manager.cc:1929] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:47.951691 19264 log.cc:1199] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/1d95d6e48173436c8c59130464413ea1
I20260924 19:57:47.952347 19264 ts_tablet_manager.cc:1950] T 1d95d6e48173436c8c59130464413ea1 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:47.884471 17812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca37ddd22f5d4bcab2862553408447ce" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:47.954860 17812 raft_consensus.cc:2468] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:47.956864 19278 tablet.cc:1751] T f43f2a255c644ca6a68a8d867798fc1a P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:57:47.977046 17817 raft_consensus.cc:2468] T bafaab10ce564b1b97659eccc0ab0967 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:47.979627 19259 raft_consensus.cc:493] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.980089 19259 raft_consensus.cc:515] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.982009 19259 leader_election.cc:290] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.982877 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 1d95d6e48173436c8c59130464413ea1 (table default.Table_a2284f2fa98d4c52949c5ecf3ce1b1ba [id=e45544040e12442cb5469c6d702695e8]) successfully deleted
I20260924 19:57:47.980331 19262 raft_consensus.cc:493] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:47.984290 19262 raft_consensus.cc:515] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:47.986109 19262 leader_election.cc:290] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:47.989573 17815 raft_consensus.cc:2468] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:47.852653 18224 tablet_bootstrap.cc:654] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:47.948064 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1137318a9fa4a0781f718dc34c4ecbe" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:47.961664 18220 tablet_bootstrap.cc:492] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:48.005962 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d129e0979a584f40ba9a975349cf15a5" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:48.008685 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89646893588340038f30ed4754623ec2" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:48.049872 18220 ts_tablet_manager.cc:1403] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.119s user 0.010s sys 0.000s
I20260924 19:57:48.052503 18220 raft_consensus.cc:359] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.053165 18220 raft_consensus.cc:385] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.053511 18220 raft_consensus.cc:740] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.054322 18220 consensus_queue.cc:260] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.056283 18220 ts_tablet_manager.cc:1434] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 19:57:48.057219 18220 tablet_bootstrap.cc:492] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:48.058835 19286 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:38450 (request call id 242) took 1118 ms. Trace:
I20260924 19:57:48.059896 18224 tablet_bootstrap.cc:492] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:48.060299 18224 ts_tablet_manager.cc:1403] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.320s user 0.009s sys 0.000s
W20260924 19:57:48.063144 18050 leader_election.cc:343] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:48.064035 18050 leader_election.cc:343] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.064450 19280 raft_consensus.cc:493] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:48.064858 18050 leader_election.cc:343] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.064877 19280 raft_consensus.cc:515] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.066706 19280 leader_election.cc:290] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:48.069927 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae7a0660795e4e4b87d0959ca84d5d1b" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:48.070501 18113 raft_consensus.cc:2468] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:48.075054 17544 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 4 LEADER]: Rejecting Update request from peer e1ff67c5d57d474a9d92cc4c6de1287e for earlier term 3. Current term is 4. Ops: [3.322-3.322]
I20260924 19:57:48.076344 18223 tablet_bootstrap.cc:700] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent opening tablet: real 0.167s user 0.001s sys 0.001s
I20260924 19:57:48.076987 18223 tablet_bootstrap.cc:654] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.076866 19240 consensus_queue.cc:1059] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 }, Status: INVALID_TERM, Last received: 3.321, Next index: 322, Last known committed idx: 321, Time since last communication: 0.001s
I20260924 19:57:48.078739 19135 leader_election.cc:290] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:48.088224 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3192f66a492442c1bf4137cfe7e13bc9" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:48.088800 18113 raft_consensus.cc:3060] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.097106 18113 raft_consensus.cc:2468] T 3192f66a492442c1bf4137cfe7e13bc9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
W20260924 19:57:48.084919 17906 leader_election.cc:343] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: ae7a0660795e4e4b87d0959ca84d5d1b
W20260924 19:57:48.098814 17906 leader_election.cc:343] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 3192f66a492442c1bf4137cfe7e13bc9
I20260924 19:57:48.095610 18220 tablet_bootstrap.cc:654] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.102734 17905 leader_election.cc:304] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
I20260924 19:57:48.104875 19280 raft_consensus.cc:2804] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:48.105218 19280 raft_consensus.cc:493] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:48.105525 19280 raft_consensus.cc:3060] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.106474 19250 catalog_manager.cc:1308] Loaded cluster ID: 08019495472c40618d8cccdc88d885cc
I20260924 19:57:48.106776 19250 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: loading cluster ID for follower catalog manager: success
I20260924 19:57:48.109395 18224 raft_consensus.cc:359] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.110158 18224 raft_consensus.cc:385] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.110448 18224 raft_consensus.cc:740] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.111280 18224 consensus_queue.cc:260] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.113127 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfe0303596204e528585c4fe1c476d8b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:48.116128 18224 ts_tablet_manager.cc:1434] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20260924 19:57:48.118718 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fd0d25961d74b67a42323a045f06b61" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:48.121518 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fd0d25961d74b67a42323a045f06b61" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:48.121912 18224 tablet_bootstrap.cc:492] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:48.124903 19240 consensus_queue.cc:1243] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Peer 6f88e00b235f48bb82e6a3c9c9f51fa6 log is divergent from this leader: its last log entry 4.322 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 321
I20260924 19:57:48.126900 19262 consensus_queue.cc:1048] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:48.133096 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfe0303596204e528585c4fe1c476d8b" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:48.135612 19308 raft_consensus.cc:3055] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 3 LEADER]: Stepping down as leader of term 3
I20260924 19:57:48.059250 19286 rpcz_store.cc:276] 0924 19:57:46.940247 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:46.943527 (+ 3280us) service_pool.cc:224] Handling call
0924 19:57:48.058760 (+1115233us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:47.054579 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0924 19:57:47.054924 (+ 345us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0924 19:57:47.054954 (+ 30us) alter_schema_op.cc:66] Acquired schema lock
0924 19:57:47.055002 (+ 48us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0924 19:57:47.055125 (+ 123us) alter_schema_op.cc:128] START. Timestamp: 1790279867055046
0924 19:57:47.055142 (+ 17us) op_driver.cc:348] REPLICATION: starting
0924 19:57:47.055696 (+ 554us) log.cc:844] Serialized 173 byte log entry
0924 19:57:47.638770 (+583074us) op_driver.cc:464] REPLICATION: finished
0924 19:57:47.693206 (+ 54436us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0924 19:57:47.733318 (+ 40112us) log.cc:844] Serialized 5 byte log entry
0924 19:57:47.916185 (+182867us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:47.916816 (+ 631us) log.cc:844] Serialized 14 byte log entry
0924 19:57:48.058618 (+141802us) alter_schema_op.cc:73] Released schema lock
0924 19:57:48.058651 (+ 33us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54258,"mutex_wait_us":28887,"prepare.queue_time_us":243,"prepare.run_cpu_time_us":1472,"prepare.run_wall_time_us":55755,"raft.queue_time_us":91567,"raft.run_cpu_time_us":19,"raft.run_wall_time_us":19,"replication_time_us":583504,"thread_start_us":55011,"threads_started":1}]]}
I20260924 19:57:48.140050 19308 raft_consensus.cc:740] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e1ff67c5d57d474a9d92cc4c6de1287e, State: Running, Role: LEADER
I20260924 19:57:48.141383 19308 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 3.322, Last appended by leader: 322, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:48.142587 19308 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 3 FOLLOWER]: Advancing to term 4
I20260924 19:57:48.144677 18224 tablet_bootstrap.cc:654] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.161616 18051 leader_election.cc:304] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.162647 19263 raft_consensus.cc:2804] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:48.163115 19263 raft_consensus.cc:697] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:48.163985 19263 consensus_queue.cc:237] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:48.169983 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 67319us
I20260924 19:57:48.173051 18224 tablet_bootstrap.cc:492] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:48.173444 18224 ts_tablet_manager.cc:1403] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.000s
I20260924 19:57:48.173955 18051 leader_election.cc:304] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.175028 19204 raft_consensus.cc:2804] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:48.175738 19204 raft_consensus.cc:697] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:48.176574 19204 consensus_queue.cc:237] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.179865 17905 leader_election.cc:304] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
I20260924 19:57:48.185545 19297 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260924 19:57:48.200985 18224 raft_consensus.cc:359] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.201654 18224 raft_consensus.cc:385] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.201897 18224 raft_consensus.cc:740] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.211094 18223 tablet_bootstrap.cc:492] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:48.212132 18224 consensus_queue.cc:260] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.181155 19280 raft_consensus.cc:515] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.202517 18220 tablet_bootstrap.cc:492] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:48.215610 19280 leader_election.cc:290] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:48.216130 18220 ts_tablet_manager.cc:1403] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.159s user 0.011s sys 0.000s
W20260924 19:57:48.218243 17906 leader_election.cc:343] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: ae7a0660795e4e4b87d0959ca84d5d1b
I20260924 19:57:48.219449 19260 raft_consensus.cc:2804] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:48.220153 18224 ts_tablet_manager.cc:1434] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20260924 19:57:48.225003 18224 tablet_bootstrap.cc:492] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:48.225481 18220 raft_consensus.cc:359] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.226166 18220 raft_consensus.cc:385] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.226434 18220 raft_consensus.cc:740] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.227075 18220 consensus_queue.cc:260] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.229182 18220 ts_tablet_manager.cc:1434] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260924 19:57:48.230011 18220 tablet_bootstrap.cc:492] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:48.250847 19260 raft_consensus.cc:697] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:48.252192 19260 consensus_queue.cc:237] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:48.257975 17753 leader_election.cc:343] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:48.258822 17753 leader_election.cc:343] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:48.259644 17753 leader_election.cc:343] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:48.260455 17753 leader_election.cc:343] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.260769 17753 leader_election.cc:304] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.261569 18224 tablet_bootstrap.cc:654] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.272987 19250 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: acquiring CA information for follower catalog manager: success
I20260924 19:57:48.276206 18938 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 4 FOLLOWER]: Refusing update from remote peer 6f88e00b235f48bb82e6a3c9c9f51fa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 249. Preceding OpId from leader: term: 4 index: 322. (index mismatch)
I20260924 19:57:48.278033 19284 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 249, Time since last communication: 0.000s
I20260924 19:57:48.216640 18051 leader_election.cc:304] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:48.283383 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 66816.6us
I20260924 19:57:48.284413 18223 ts_tablet_manager.cc:1403] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.380s user 0.009s sys 0.001s
W20260924 19:57:48.261696 17753 leader_election.cc:343] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.291289 19204 raft_consensus.cc:2804] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:48.291363 17753 leader_election.cc:304] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.291652 19204 raft_consensus.cc:493] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:48.291965 19204 raft_consensus.cc:3060] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.292488 19303 raft_consensus.cc:2749] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:48.293988 17753 leader_election.cc:343] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.294354 17753 leader_election.cc:304] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.263023 19213 raft_consensus.cc:2749] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:48.299261 19303 raft_consensus.cc:2749] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:48.306502 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae7a0660795e4e4b87d0959ca84d5d1b" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:48.307276 18220 tablet_bootstrap.cc:654] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.308003 18113 raft_consensus.cc:3060] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.324442 18220 tablet_bootstrap.cc:492] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:48.325003 18220 ts_tablet_manager.cc:1403] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.005s
I20260924 19:57:48.327481 18220 raft_consensus.cc:359] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.328142 18220 raft_consensus.cc:385] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.328413 18220 raft_consensus.cc:740] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.329042 18220 consensus_queue.cc:260] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.331174 18220 ts_tablet_manager.cc:1434] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260924 19:57:48.332090 18220 tablet_bootstrap.cc:492] T d129e0979a584f40ba9a975349cf15a5 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:48.338045 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1137318a9fa4a0781f718dc34c4ecbe" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:48.338713 17810 raft_consensus.cc:2468] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:48.339893 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d129e0979a584f40ba9a975349cf15a5" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:48.340540 17810 raft_consensus.cc:2468] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:48.344262 18220 tablet_bootstrap.cc:654] T d129e0979a584f40ba9a975349cf15a5 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.344398 17809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89646893588340038f30ed4754623ec2" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:48.345000 17809 raft_consensus.cc:2468] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:48.353881 18224 tablet_bootstrap.cc:492] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:48.354419 18224 ts_tablet_manager.cc:1403] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.130s user 0.014s sys 0.002s
I20260924 19:57:48.358686 18220 tablet_bootstrap.cc:492] T d129e0979a584f40ba9a975349cf15a5 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:48.359081 18220 ts_tablet_manager.cc:1403] T d129e0979a584f40ba9a975349cf15a5 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20260924 19:57:48.286724 18223 raft_consensus.cc:359] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.359495 18223 raft_consensus.cc:385] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.359778 18223 raft_consensus.cc:740] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.360540 18223 consensus_queue.cc:260] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.362476 18223 ts_tablet_manager.cc:1434] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.078s user 0.005s sys 0.000s
I20260924 19:57:48.368283 18220 raft_consensus.cc:359] T d129e0979a584f40ba9a975349cf15a5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.368911 18220 raft_consensus.cc:385] T d129e0979a584f40ba9a975349cf15a5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.369155 18220 raft_consensus.cc:740] T d129e0979a584f40ba9a975349cf15a5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.369702 18220 consensus_queue.cc:260] T d129e0979a584f40ba9a975349cf15a5 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.362223 18224 raft_consensus.cc:359] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.377596 18220 ts_tablet_manager.cc:1434] T d129e0979a584f40ba9a975349cf15a5 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260924 19:57:48.378460 18220 tablet_bootstrap.cc:492] T 57008266b80345e0a3f8e60bd7e2d62b P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:48.376039 18224 raft_consensus.cc:385] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.378948 18224 raft_consensus.cc:740] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.382082 18224 consensus_queue.cc:260] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.384790 19260 raft_consensus.cc:493] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:48.385409 19260 raft_consensus.cc:515] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.398589 18051 leader_election.cc:304] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.399840 19263 raft_consensus.cc:2804] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:48.400414 19263 raft_consensus.cc:493] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:48.400888 19263 raft_consensus.cc:3060] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.403507 18051 leader_election.cc:304] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.407250 18051 leader_election.cc:304] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.408507 19080 raft_consensus.cc:2804] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:48.408824 19080 raft_consensus.cc:493] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:48.409127 19080 raft_consensus.cc:3060] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.398236 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca37ddd22f5d4bcab2862553408447ce" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:48.410465 17821 raft_consensus.cc:2468] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:48.402729 19213 raft_consensus.cc:493] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:48.411475 19213 raft_consensus.cc:515] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.413193 18224 ts_tablet_manager.cc:1434] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.058s user 0.005s sys 0.000s
I20260924 19:57:48.413458 19213 leader_election.cc:290] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:48.414111 18224 tablet_bootstrap.cc:492] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:48.415796 17625 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 4 FOLLOWER]: Refusing update from remote peer 6f88e00b235f48bb82e6a3c9c9f51fa6: Log matching property violated. Preceding OpId in replica: term: 3 index: 322. Preceding OpId from leader: term: 4 index: 322. (term mismatch)
I20260924 19:57:48.416162 17625 pending_rounds.cc:85] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Aborting all ops after (but not including) 321
I20260924 19:57:48.416280 17906 leader_election.cc:304] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:48.416422 17625 pending_rounds.cc:107] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Aborting uncommitted WRITE_OP operation due to leader change: 3.322
I20260924 19:57:48.417037 19315 raft_consensus.cc:2804] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:48.417335 19315 raft_consensus.cc:493] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:48.417625 19315 raft_consensus.cc:3060] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
E20260924 19:57:48.419130 17671 catalog_manager.cc:839] Error processing pending assignments: Aborted: error persisting updated tablet metadata: Op aborted by new leader
I20260924 19:57:48.419282 19262 raft_consensus.cc:2804] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:48.419610 19262 raft_consensus.cc:493] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:48.419911 19262 raft_consensus.cc:3060] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.419767 19284 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20260924 19:57:48.426591 19308 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f88e00b235f48bb82e6a3c9c9f51fa6. Latest consensus state: current_term: 4 leader_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:48.427434 19308 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:48.456086 18220 tablet_bootstrap.cc:654] T 57008266b80345e0a3f8e60bd7e2d62b P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.457744 17610 catalog_manager.cc:5838] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:48.465713 17610 catalog_manager.cc:5838] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:48.472273 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9f9b76a58e47c0beee5ce94a44b707" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:48.479943 18220 tablet_bootstrap.cc:492] T 57008266b80345e0a3f8e60bd7e2d62b P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:48.480444 18220 ts_tablet_manager.cc:1403] T 57008266b80345e0a3f8e60bd7e2d62b P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.102s user 0.010s sys 0.000s
I20260924 19:57:48.458391 19260 leader_election.cc:290] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:48.477293 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:48.489650 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet f43f2a255c644ca6a68a8d867798fc1a (table default.Table_4e3a3158cde24302806b7ba7827831c4 [id=fa0b10df171340868b0305a28d192ea5]). Expected version 1 got 0
I20260924 19:57:48.491928 18220 raft_consensus.cc:359] T 57008266b80345e0a3f8e60bd7e2d62b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.494896 18220 raft_consensus.cc:385] T 57008266b80345e0a3f8e60bd7e2d62b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:57:48.496021 17753 leader_election.cc:343] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.496778 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:48.496165 18220 raft_consensus.cc:740] T 57008266b80345e0a3f8e60bd7e2d62b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.498190 18220 consensus_queue.cc:260] T 57008266b80345e0a3f8e60bd7e2d62b P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.504189 18220 ts_tablet_manager.cc:1434] T 57008266b80345e0a3f8e60bd7e2d62b P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260924 19:57:48.502022 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]). Expected version 1 got 0
I20260924 19:57:48.507118 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9f9b76a58e47c0beee5ce94a44b707" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:48.507189 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca37ddd22f5d4bcab2862553408447ce" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:48.508888 18223 tablet_bootstrap.cc:492] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:48.508901 17600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:57:48.510697 18224 tablet_bootstrap.cc:654] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.516201 19250 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:57:48.519094 18220 tablet_bootstrap.cc:492] T bafaab10ce564b1b97659eccc0ab0967 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:48.506319 17610 catalog_manager.cc:5838] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:57:48.527242 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 3 got 0
I20260924 19:57:48.430265 19315 raft_consensus.cc:515] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:48.531716 17753 leader_election.cc:343] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.531416 17610 catalog_manager.cc:5838] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:48.532160 17753 leader_election.cc:304] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.532927 19213 raft_consensus.cc:2749] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:48.536147 17671 catalog_manager.cc:1651] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:57:48.539496 18224 tablet_bootstrap.cc:492] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:48.539952 18224 ts_tablet_manager.cc:1403] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.126s user 0.006s sys 0.004s
I20260924 19:57:48.541944 18113 raft_consensus.cc:2468] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:48.542557 18224 raft_consensus.cc:359] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.543403 18224 raft_consensus.cc:385] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.543668 18224 raft_consensus.cc:740] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.544322 18224 consensus_queue.cc:260] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.554314 18223 tablet_bootstrap.cc:654] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.556818 18224 ts_tablet_manager.cc:1434] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.017s user 0.005s sys 0.002s
I20260924 19:57:48.557768 18224 tablet_bootstrap.cc:492] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:48.560777 19308 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:48.561468 19308 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:48.575709 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca37ddd22f5d4bcab2862553408447ce" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:48.576238 17815 raft_consensus.cc:3060] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.582685 17815 raft_consensus.cc:2468] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:48.590224 19284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:48.590947 19284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:48.556905 17905 leader_election.cc:304] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202
I20260924 19:57:48.591745 18220 tablet_bootstrap.cc:654] T bafaab10ce564b1b97659eccc0ab0967 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.592054 19135 raft_consensus.cc:2804] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:48.592509 19135 raft_consensus.cc:697] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:48.593300 19135 consensus_queue.cc:237] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.583859 17906 leader_election.cc:304] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:48.594494 18224 tablet_bootstrap.cc:654] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.595034 19280 raft_consensus.cc:2804] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:48.565872 19260 raft_consensus.cc:493] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:48.606716 19260 raft_consensus.cc:515] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.611406 19260 leader_election.cc:290] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:48.611830 19213 raft_consensus.cc:493] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:48.612295 19213 raft_consensus.cc:515] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.612855 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431d298f513d47268284d50af2ea1436" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:48.613718 17810 raft_consensus.cc:2393] T 431d298f513d47268284d50af2ea1436 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:48.614267 19213 leader_election.cc:290] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:48.616039 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96379e7356ab40218c6ee2c126c0b4ae" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:48.617240 17753 leader_election.cc:343] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.589093 19315 leader_election.cc:290] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:48.625463 18224 tablet_bootstrap.cc:492] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:48.625957 18224 ts_tablet_manager.cc:1403] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.000s
E20260924 19:57:48.561861 17610 catalog_manager.cc:6001] Error updating tablets from eaf64bfb64aa42948c9bc1411ce4ce35: Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "3192f66a492442c1bf4137cfe7e13bc9" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ae7a0660795e4e4b87d0959ca84d5d1b" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "431d298f513d47268284d50af2ea1436" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f169c47d711d4566b2004cdfd0de725d" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "0b99ff1920274831b58d8283bd592818" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "edc6c16578e5463a9d44c25fd6c538a9" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27728 live_row_count: 0 } } updated_tablets { tablet_id: "92c0daa7d67e4aa390e334c669adda76" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27428 live_row_count: 0 } } updated_tablets { tablet_id: "201ea0020cb3430eac85e80b8c06fb07" state: STOPPED consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_DELETED } updated_tablets { tablet_id: "5df3dcf4ddcf4efa9aa047da56dfb7a7" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27506 live_row_count: 0 } } updated_tablets { tablet_id: "924bc897f48b44069f1fb2ee71e26db7" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "affce4de8e0b412aa3b4bf19f9be7b6e" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27506 live_row_count: 0 } } updated_tablets { tablet_id: "e0246d83189d4ba5b75fbd8390f517f6" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27506 live_row_count: 0 } } updated_tablets { tablet_id: "077f70bcee7e412299da12ddc30a63ff" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } updated_tablets { tablet_id: "954178b8cc1e434085337f66b0e85672" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 3 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "8aef0359a77240679240823612507ee8" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "0c08f10d2a364cb8a0a7ea40654c3fe4" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } updated_tablets { tablet_id: "274f3decdc7e42c0917bf28151c4da71" state: RUNNING consensus_state { current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "3b328d37d55e4116a29322caaa21d431" state: RUNNING consensus_state { current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6b7f2dd8ba2a4cb190e9ff92a542bacb" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "c1684aaf9d984649bff8c4b0b12044b9" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27759 live_row_count: 0 } } updated_tablets { tablet_id: "2e6d926ff7c24901a15b0c7b2239fbad" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27506 live_row_count: 0 } } updated_tablets { tablet_id: "97a1b8ccd35a43cea7a71d705fad1494" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6bd442f16492495f84bb8363b9bade97" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } updated_tablets { tablet_id: "dbb82914df65416aaf47db8e507afb40" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27728 live_row_count: 0 } } updated_tablets { tablet_id: "dd434e38c9364f55bd18383149015028" state: RUNNING consensus_state { current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "fd6e656a653a43d1843a91fee813237d" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f43f2a255c644ca6a68a8d867798fc1a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "7c7b8422753a4746bde71fcb25409949" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e68378312c334fc28dafb71bf6386be8" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } updated_tablets { tablet_id: "c0feecd2fdb4409f86f6a5fbfbab3537" state: RUNNING consensus_state { current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "872bc32b9e384c828a0699079b27530d" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } updated_tablets { tablet_id: "0cf2df461598411f8ad9f3a51eafc6f7" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "484517ee7df8463bb70d201ead1c6835" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "df722b0bbfd04ce2943f0e5b8058c50f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } removed_tablet_ids: "c884f12860164f12bbcd0d2f87067013" removed_tablet_ids: "ba46ac86b242407abd3e8dbf09f12efa" removed_tablet_ids: "151247d6b5254cd884c58db9311ac04a" removed_tablet_ids: "a765abd702644f079bddffb3fc4238ef" removed_tablet_ids: "dd2f4eb8bf254c4e85b6a9072271766b" removed_tablet_ids: "2cf62a73c49245198d65011875148dc3" removed_tablet_ids: "07f86f24d9d14d05898e1c0f69428850" removed_tablet_ids: "d970cc92a1134a98a613d198a319cb01" removed_tablet_ids: "910d61b8c3574302b5a328187226571e" removed_tablet_ids: "f482b53a4a2d44f6b3431c588cd605ce" removed_tablet_ids: "fec9b6ecfb384b528cf71c4cfb2fd4cb" removed_tablet_ids: "41bdb846ca19425b882434df0167479e" removed_tablet_ids: "8a524bad1c4040e58ad84fcf5a4fc47d" removed_tablet_ids: "83ee43b6dc064dc7b320691d52e57ea7" removed_tablet_ids: "24f84032bca24a7ca9da65f0bbceb0d5" removed_tablet_ids: "349c6e9413bb41268d0c41d549203e2a" removed_tablet_ids: "9c861e93ad9f40ba9ad010021e43bc2b" removed_tablet_ids: "ef3f82de70bb4328a15a044fe0e500bf" removed_tablet_ids: "32466931871c4c5d8e9f95a11ffe26be" removed_tablet_ids: "44199d2655f14dcf97afe2180789aae4" removed_tablet_ids: "5f994e0401df4fc19bc5abb10f08bc6e" removed_tablet_ids: "efd7339d0fee436299a7eac02a82addd" removed_tablet_ids: "a2e4519aed7642678db3a518a89778de" sequence_number: 34
I20260924 19:57:48.635131 17971 raft_consensus.cc:3060] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.642343 19280 raft_consensus.cc:697] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:48.643674 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:48.643536 19280 consensus_queue.cc:237] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.645465 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:48.645504 18224 raft_consensus.cc:359] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.646130 18224 raft_consensus.cc:385] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.646374 18224 raft_consensus.cc:740] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.647944 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 3 got 0
I20260924 19:57:48.545367 19204 raft_consensus.cc:515] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.648964 18111 raft_consensus.cc:2393] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:48.652844 19204 leader_election.cc:290] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:48.579291 19080 raft_consensus.cc:515] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.580380 19263 raft_consensus.cc:515] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.654986 19080 leader_election.cc:290] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:48.655210 18224 consensus_queue.cc:260] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.655826 19263 leader_election.cc:290] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:48.589586 19262 raft_consensus.cc:515] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.661698 18224 ts_tablet_manager.cc:1434] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20260924 19:57:48.661680 19262 leader_election.cc:290] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:48.662647 18224 tablet_bootstrap.cc:492] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:48.593933 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca37ddd22f5d4bcab2862553408447ce" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:48.667589 18113 raft_consensus.cc:2393] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:48.657943 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca37ddd22f5d4bcab2862553408447ce" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:48.659008 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d129e0979a584f40ba9a975349cf15a5" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:48.668733 17821 raft_consensus.cc:3060] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.659334 17812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1137318a9fa4a0781f718dc34c4ecbe" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:48.671633 17812 raft_consensus.cc:3060] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.605562 19329 consensus_queue.cc:1048] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:48.622879 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96379e7356ab40218c6ee2c126c0b4ae" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:48.627259 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431d298f513d47268284d50af2ea1436" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:48.673452 19135 consensus_queue.cc:1048] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:48.677136 18220 tablet_bootstrap.cc:492] T bafaab10ce564b1b97659eccc0ab0967 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:48.677565 18220 ts_tablet_manager.cc:1403] T bafaab10ce564b1b97659eccc0ab0967 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.159s user 0.012s sys 0.001s
I20260924 19:57:48.679235 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca37ddd22f5d4bcab2862553408447ce" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:48.679952 18220 raft_consensus.cc:359] T bafaab10ce564b1b97659eccc0ab0967 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.680671 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d129e0979a584f40ba9a975349cf15a5" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:48.681160 17972 raft_consensus.cc:3060] T d129e0979a584f40ba9a975349cf15a5 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:48.680564 18220 raft_consensus.cc:385] T bafaab10ce564b1b97659eccc0ab0967 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.681738 18220 raft_consensus.cc:740] T bafaab10ce564b1b97659eccc0ab0967 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.682350 18220 consensus_queue.cc:260] T bafaab10ce564b1b97659eccc0ab0967 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.684321 18220 ts_tablet_manager.cc:1434] T bafaab10ce564b1b97659eccc0ab0967 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260924 19:57:48.685158 18220 tablet_bootstrap.cc:492] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:48.687491 17810 raft_consensus.cc:2393] T ca37ddd22f5d4bcab2862553408447ce P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eaf64bfb64aa42948c9bc1411ce4ce35 in current term 1: Already voted for candidate f74e6ed351284d7fb110e44336ce643a in this term.
I20260924 19:57:48.691649 17821 raft_consensus.cc:2468] T d129e0979a584f40ba9a975349cf15a5 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:48.692085 19303 raft_consensus.cc:493] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:48.692303 19213 raft_consensus.cc:493] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:48.648027 17610 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 36) took 1974 ms (client timeout 1000 ms). Trace:
I20260924 19:57:48.693935 17972 raft_consensus.cc:2468] T d129e0979a584f40ba9a975349cf15a5 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:48.694607 17608 catalog_manager.cc:5938] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301) does not have the latest schema for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]). Expected version 1 got 0
W20260924 19:57:48.693836 17610 rpcz_store.cc:269] 0924 19:57:46.673982 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:46.674130 (+ 148us) service_pool.cc:224] Handling call
0924 19:57:48.561583 (+1887453us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:48.561614 (+ 31us) write_op.cc:454] Released schema lock
0924 19:57:48.647952 (+ 86338us) inbound_call.cc:177] Queueing failure response
Related trace 'op':
0924 19:57:48.561309 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER)
0924 19:57:48.561365 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:48.561397 (+ 32us) write_op.cc:454] Released schema lock
0924 19:57:48.561426 (+ 29us) write_op.cc:335] FINISH: Op aborted
Metrics: {"reported_tablets":34,"child_traces":[["op",{}]]}
I20260924 19:57:48.692548 19303 raft_consensus.cc:515] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.698513 19303 leader_election.cc:290] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:48.692692 19213 raft_consensus.cc:515] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.700803 19213 leader_election.cc:290] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:48.705964 18224 tablet_bootstrap.cc:654] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.710753 18220 tablet_bootstrap.cc:654] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.712363 17608 catalog_manager.cc:5838] T 274f3decdc7e42c0917bf28151c4da71 P f74e6ed351284d7fb110e44336ce643a reported cstate change: term changed from 1 to 2, leader changed from eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131) to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 2 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } }
I20260924 19:57:48.720399 18051 leader_election.cc:304] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
W20260924 19:57:48.724777 17753 leader_election.cc:343] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.725193 17753 leader_election.cc:304] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:48.725364 17610 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 37) took 940 ms (client timeout 1000 ms). Trace:
W20260924 19:57:48.725837 17610 rpcz_store.cc:269] 0924 19:57:47.785302 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:48.695641 (+910339us) service_pool.cc:224] Handling call
0924 19:57:48.725295 (+ 29654us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:48.729809 19329 raft_consensus.cc:2804] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:48.730780 19329 raft_consensus.cc:697] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:48.725946 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25dd98712d54cd5b53df00b2cf9f3c4" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:48.727268 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1137318a9fa4a0781f718dc34c4ecbe" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:48.728363 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89646893588340038f30ed4754623ec2" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:48.731767 19329 consensus_queue.cc:237] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.738715 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335f979eb04c49bcb53023693f54ad41" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:48.739046 18115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25dd98712d54cd5b53df00b2cf9f3c4" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:48.750109 17905 leader_election.cc:304] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:48.750803 19135 raft_consensus.cc:2749] T 431d298f513d47268284d50af2ea1436 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:48.753880 18223 tablet_bootstrap.cc:492] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:48.754290 18223 ts_tablet_manager.cc:1403] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.246s user 0.010s sys 0.005s
I20260924 19:57:48.756790 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89646893588340038f30ed4754623ec2" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:48.757025 18220 tablet_bootstrap.cc:492] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
W20260924 19:57:48.757125 17753 leader_election.cc:343] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.757372 17821 raft_consensus.cc:3060] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:48.757987 17753 leader_election.cc:343] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:48.758484 17753 leader_election.cc:343] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.764263 17821 raft_consensus.cc:2468] T 89646893588340038f30ed4754623ec2 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:48.769721 18050 leader_election.cc:304] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.770568 19262 raft_consensus.cc:2749] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 19:57:48.772562 19262 consensus_queue.cc:1048] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:48.759719 19213 raft_consensus.cc:2749] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:48.775286 17812 raft_consensus.cc:2468] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:48.779613 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335f979eb04c49bcb53023693f54ad41" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:48.781478 18050 leader_election.cc:343] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:48.782244 18050 leader_election.cc:343] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.758780 17753 leader_election.cc:304] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.802023 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_65d20bb187ea4069b2e589e7643028be [id=00e4a8cdcb374ae6a4446b3b7404d7a8]
W20260924 19:57:48.802840 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 44369.3us
W20260924 19:57:48.806382 17610 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 46) took 770 ms (client timeout 1000 ms). Trace:
W20260924 19:57:48.806725 17610 rpcz_store.cc:269] 0924 19:57:48.030340 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:48.763517 (+733177us) service_pool.cc:224] Handling call
0924 19:57:48.801116 (+ 37599us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:48.807823 19135 consensus_queue.cc:1048] T 924bc897f48b44069f1fb2ee71e26db7 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:48.803309 19341 raft_consensus.cc:2749] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:48.808398 18224 tablet_bootstrap.cc:492] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:48.811440 18224 ts_tablet_manager.cc:1403] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.149s user 0.006s sys 0.004s
I20260924 19:57:48.812095 18051 leader_election.cc:304] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.813241 19329 raft_consensus.cc:2804] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:48.813735 18224 raft_consensus.cc:359] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.814327 18224 raft_consensus.cc:385] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.814594 18224 raft_consensus.cc:740] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.814241 19329 raft_consensus.cc:697] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:48.815701 19329 consensus_queue.cc:237] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.817005 18223 raft_consensus.cc:359] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.817685 18223 raft_consensus.cc:385] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.817957 18223 raft_consensus.cc:740] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:48.824908 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_5cc282e71cd74150a849aff1620d54f1 [id=010f7a5a46034f008e83a6492524b778]
E20260924 19:57:48.716501 17608 catalog_manager.cc:6001] Error updating tablets from f74e6ed351284d7fb110e44336ce643a: Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "f169c47d711d4566b2004cdfd0de725d" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "44199d2655f14dcf97afe2180789aae4" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "274f3decdc7e42c0917bf28151c4da71" state: RUNNING consensus_state { current_term: 2 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27550 live_row_count: 0 } } updated_tablets { tablet_id: "affce4de8e0b412aa3b4bf19f9be7b6e" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "3b328d37d55e4116a29322caaa21d431" state: RUNNING consensus_state { current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "0c08f10d2a364cb8a0a7ea40654c3fe4" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "92c0daa7d67e4aa390e334c669adda76" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d81f74c7ba2443e49228b17e867588cc" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } updated_tablets { tablet_id: "edc6c16578e5463a9d44c25fd6c538a9" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6b7f2dd8ba2a4cb190e9ff92a542bacb" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } updated_tablets { tablet_id: "872bc32b9e384c828a0699079b27530d" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "0cf2df461598411f8ad9f3a51eafc6f7" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } updated_tablets { tablet_id: "c0feecd2fdb4409f86f6a5fbfbab3537" state: RUNNING consensus_state { current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "954178b8cc1e434085337f66b0e85672" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } } } schema_version: 3 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 28284 live_row_count: 0 } } updated_tablets { tablet_id: "e0246d83189d4ba5b75fbd8390f517f6" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "5df3dcf4ddcf4efa9aa047da56dfb7a7" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2e6d926ff7c24901a15b0c7b2239fbad" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "c1684aaf9d984649bff8c4b0b12044b9" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6bd442f16492495f84bb8363b9bade97" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "fd6e656a653a43d1843a91fee813237d" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } updated_tablets { tablet_id: "e68378312c334fc28dafb71bf6386be8" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ba46ac86b242407abd3e8dbf09f12efa" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "7c7b8422753a4746bde71fcb25409949" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27537 live_row_count: 0 } } updated_tablets { tablet_id: "077f70bcee7e412299da12ddc30a63ff" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "97a1b8ccd35a43cea7a71d705fad1494" state: RUNNING consensus_state { current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27759 live_row_count: 0 } } updated_tablets { tablet_id: "dbb82914df65416aaf47db8e507afb40" state: RUNNING consensus_state { current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "dd434e38c9364f55bd18383149015028" state: RUNNING consensus_state { current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } removed_tablet_ids: "c884f12860164f12bbcd0d2f87067013" removed_tablet_ids: "d970cc92a1134a98a613d198a319cb01" removed_tablet_ids: "9b04059219084f608e8cb5fa4d0e0d58" removed_tablet_ids: "a765abd702644f079bddffb3fc4238ef" removed_tablet_ids: "2cf62a73c49245198d65011875148dc3" removed_tablet_ids: "41bdb846ca19425b882434df0167479e" removed_tablet_ids: "910d61b8c3574302b5a328187226571e" removed_tablet_ids: "24f84032bca24a7ca9da65f0bbceb0d5" removed_tablet_ids: "07f86f24d9d14d05898e1c0f69428850" removed_tablet_ids: "151247d6b5254cd884c58db9311ac04a" removed_tablet_ids: "201ea0020cb3430eac85e80b8c06fb07" removed_tablet_ids: "f482b53a4a2d44f6b3431c588cd605ce" removed_tablet_ids: "f38188eccc344246b72a8755ec5201ed" removed_tablet_ids: "9c2440e1cade4327a63d6234e3adcd64" removed_tablet_ids: "ef3f82de70bb4328a15a044fe0e500bf" removed_tablet_ids: "dd2f4eb8bf254c4e85b6a9072271766b" removed_tablet_ids: "085b8088134d47478f0ce06a6a9f8572" removed_tablet_ids: "349c6e9413bb41268d0c41d549203e2a" removed_tablet_ids: "8a524bad1c4040e58ad84fcf5a4fc47d" removed_tablet_ids: "f90a677e94034c73ab5452554862c1c3" removed_tablet_ids: "8a19b5812dc54589a5cbf48199951600" removed_tablet_ids: "a2e4519aed7642678db3a518a89778de" removed_tablet_ids: "a384c9a0bb65429ebb97fe0807c78773" removed_tablet_ids: "9c861e93ad9f40ba9ad010021e43bc2b" removed_tablet_ids: "916877c7540a4f6685064fc2807cb140" removed_tablet_ids: "dd2f548d5bd5454cb16335473a9eeb03" removed_tablet_ids: "efd7339d0fee436299a7eac02a82addd" removed_tablet_ids: "fec9b6ecfb384b528cf71c4cfb2fd4cb" removed_tablet_ids: "5f994e0401df4fc19bc5abb10f08bc6e" removed_tablet_ids: "83ee43b6dc064dc7b320691d52e57ea7" sequence_number: 43
I20260924 19:57:48.818625 18223 consensus_queue.cc:260] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.842223 18224 consensus_queue.cc:260] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.844169 18224 ts_tablet_manager.cc:1434] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260924 19:57:48.844571 18223 ts_tablet_manager.cc:1434] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.090s user 0.006s sys 0.000s
I20260924 19:57:48.845110 18224 tablet_bootstrap.cc:492] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:48.848956 18051 leader_election.cc:304] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.850214 19262 raft_consensus.cc:2804] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:48.850669 19262 raft_consensus.cc:697] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:48.851430 18220 ts_tablet_manager.cc:1403] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.166s user 0.011s sys 0.002s
I20260924 19:57:48.851482 19262 consensus_queue.cc:237] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.853881 18220 raft_consensus.cc:359] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.854535 18220 raft_consensus.cc:385] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:48.854797 18220 raft_consensus.cc:740] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
W20260924 19:57:48.858187 17753 leader_election.cc:343] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.858605 17753 leader_election.cc:304] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.859447 19341 raft_consensus.cc:2749] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:48.881682 18223 tablet_bootstrap.cc:492] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:48.891698 18224 tablet_bootstrap.cc:654] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:48.894083 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_e91b551877474bb0914b854a58447b3a [id=0606053a508145868b640f364fbf48d6]
W20260924 19:57:48.894632 17608 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 45) took 2057 ms (client timeout 1000 ms). Trace:
I20260924 19:57:48.910233 18220 consensus_queue.cc:260] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.912542 18220 ts_tablet_manager.cc:1434] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.061s user 0.006s sys 0.000s
I20260924 19:57:48.913515 18220 tablet_bootstrap.cc:492] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:48.920081 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_db4a5e39fd954bb0b4c271899feefe90 [id=0b7e96c71a074f838a519746d9c92c3f]
W20260924 19:57:48.894999 17608 rpcz_store.cc:269] 0924 19:57:46.837376 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:47.140460 (+303084us) service_pool.cc:224] Handling call
0924 19:57:48.716283 (+1575823us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:48.716303 (+ 20us) write_op.cc:454] Released schema lock
0924 19:57:48.894554 (+178251us) inbound_call.cc:177] Queueing failure response
Related trace 'op':
0924 19:57:48.716088 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER)
0924 19:57:48.716122 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:48.716144 (+ 22us) write_op.cc:454] Released schema lock
0924 19:57:48.716166 (+ 22us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":34135,"reported_tablets":27,"child_traces":[["op",{}]]}
I20260924 19:57:48.926867 19213 raft_consensus.cc:493] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:48.927356 19213 raft_consensus.cc:515] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:48.926916 19341 raft_consensus.cc:493] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:48.928982 19341 raft_consensus.cc:515] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:48.930842 19341 leader_election.cc:290] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:48.935405 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c1afbfc7de7480ab4be40680a3e3ada" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:48.936493 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c1afbfc7de7480ab4be40680a3e3ada" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:48.936712 17753 leader_election.cc:343] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.938762 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_081e10d31dc74797abba37099d8a75fc [id=0ea22c5631c347aa9c7aa0e24140cb14]
W20260924 19:57:48.940856 17753 leader_election.cc:343] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.941227 17753 leader_election.cc:304] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.947522 19213 leader_election.cc:290] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:48.948619 19341 raft_consensus.cc:2749] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:48.950801 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accddc7f696142bfa6a5e655f5abc220" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:48.951853 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accddc7f696142bfa6a5e655f5abc220" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:48.952507 17610 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 49) took 834 ms (client timeout 1000 ms). Trace:
W20260924 19:57:48.955390 17753 leader_election.cc:343] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:48.955964 17753 leader_election.cc:343] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:48.956288 17753 leader_election.cc:304] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:48.960605 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_05427967e4cc480cba4d8f7128b23677 [id=11bd6526602f4c0dbdb90bbc8806262b]
W20260924 19:57:48.952847 17610 rpcz_store.cc:269] 0924 19:57:48.118120 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:48.814582 (+696462us) service_pool.cc:224] Handling call
0924 19:57:48.952436 (+137854us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:48.964743 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]). Expected version 2 got 0
I20260924 19:57:48.965965 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 78801b8d0b664217a74b20160ba2d8ac (table default.Table_454683a3d79d4988b41292f7776df549 [id=81b26242949442c0a7db6f55f0f18097]). Expected version 1 got 0
I20260924 19:57:48.978560 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_6b68af48dd4140dda7d1854a3cf2269f [id=1a5ed525f4364782a0793571d4caa5cb]
I20260924 19:57:48.979351 19213 raft_consensus.cc:2749] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:48.991321 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_39ac82df7ce14073912622fcd1b57aaa [id=1ae05845cc1641ffaf3b82b0f1390e43]
I20260924 19:57:48.994350 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_202c25f88e1a485c9ca6677d9cf53e75 [id=221fc6b9725747ada3ed73dc569052cb]
I20260924 19:57:49.004691 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_fcbe3b81791a4aaf923847b1366eb6f9 [id=25cb87a38f0f473d99bad6f06d4edcf4]
I20260924 19:57:49.007084 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_66edcc97f7a64dfa85bf24d26e49c5de [id=31d1abb2c65744d78c5e61dd4ae81b63]
I20260924 19:57:49.008769 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]
I20260924 19:57:49.010138 18223 tablet_bootstrap.cc:654] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:49.011503 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b]
I20260924 19:57:49.015604 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265]
I20260924 19:57:49.017103 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_9faf28b24a2c47449a862536ea873f36 [id=3d02208bd3604c729b474a4f005b8cf8]
I20260924 19:57:49.018623 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]
I20260924 19:57:49.020807 19204 raft_consensus.cc:493] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:49.020925 18224 tablet_bootstrap.cc:492] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:49.021370 18224 ts_tablet_manager.cc:1403] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.176s user 0.012s sys 0.007s
I20260924 19:57:49.021283 19204 raft_consensus.cc:515] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.023281 19204 leader_election.cc:290] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:49.023854 18224 raft_consensus.cc:359] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.024483 18224 raft_consensus.cc:385] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:49.024827 18224 raft_consensus.cc:740] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:49.025148 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78801b8d0b664217a74b20160ba2d8ac" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:49.027046 18050 leader_election.cc:343] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:49.027064 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78801b8d0b664217a74b20160ba2d8ac" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:49.027657 17810 raft_consensus.cc:2468] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:49.028898 18051 leader_election.cc:304] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:49.029901 19204 raft_consensus.cc:2804] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:49.030244 19204 raft_consensus.cc:493] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:49.030557 19204 raft_consensus.cc:3060] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:49.034956 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]). Expected version 1 got 0
I20260924 19:57:49.029641 18224 consensus_queue.cc:260] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.038200 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet f43f2a255c644ca6a68a8d867798fc1a (table default.Table_4e3a3158cde24302806b7ba7827831c4 [id=fa0b10df171340868b0305a28d192ea5]). Expected version 1 got 0
I20260924 19:57:49.032217 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_1dad27c8fd4445fb89e32722eef0abe2 [id=4cd61b434f884eed908abf700d02cdd0]
I20260924 19:57:49.039481 18220 tablet_bootstrap.cc:654] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:49.041849 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_b2824cbbb81c4186840b758c61eccbb2 [id=4e169b185baf4e7a81735a13b6d82fca]
I20260924 19:57:49.043334 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_e3a6abe9c34a4fd6a8567a9c0a8190c3 [id=4e80d69486a34b848393f01e656d122b]
I20260924 19:57:49.044843 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_9259880c400d44d69831fe68d31b06f4 [id=4f8c78c1109f4245b3ff41258543de1e]
I20260924 19:57:49.052297 18224 ts_tablet_manager.cc:1434] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.031s user 0.007s sys 0.001s
I20260924 19:57:49.053308 18224 tablet_bootstrap.cc:492] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:49.053366 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_9b8f6ad547fc4c3fa379c93590aebd56 [id=4fc8e565a6a640549716cee596fa3f9d]
I20260924 19:57:49.054875 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]
I20260924 19:57:49.057845 18223 tablet_bootstrap.cc:492] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:49.058275 18223 ts_tablet_manager.cc:1403] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.177s user 0.009s sys 0.000s
I20260924 19:57:49.060101 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_5e789131162643caa4c014a55244066f [id=52ff52bfd3904ab0b4826673127f70ac]
I20260924 19:57:49.060453 19308 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6f88e00b235f48bb82e6a3c9c9f51fa6)
I20260924 19:57:49.060968 19308 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:49.061638 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_5e50f3dbbb674d658a5afac0ebcedaa6 [id=545488f05d324b748e42dae4a665f0d6]
I20260924 19:57:49.064002 19308 leader_election.cc:290] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689)
I20260924 19:57:49.064227 18937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:49.067195 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 3 got 0
I20260924 19:57:49.067179 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]
I20260924 19:57:49.069118 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_818936b597264e41853b635f6232f509 [id=72c7d8cc98ee49679d8c7589a9ac0c8f]). Expected version 1 got 0
I20260924 19:57:49.070416 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet e1137318a9fa4a0781f718dc34c4ecbe (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]). Expected version 1 got 0
I20260924 19:57:49.082422 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_93373f379bb14b6ca7a3ac7a628cad63 [id=5878a9a1b1d54055a5e8dc712b9de665]
I20260924 19:57:49.086090 19320 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6f88e00b235f48bb82e6a3c9c9f51fa6)
I20260924 19:57:49.086599 19320 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:49.089169 19320 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:49.085824 19204 raft_consensus.cc:515] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.090571 19326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f88e00b235f48bb82e6a3c9c9f51fa6. Latest consensus state: current_term: 4 leader_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:49.099092 19326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:49.099960 17971 raft_consensus.cc:3060] T bafaab10ce564b1b97659eccc0ab0967 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:49.101707 18223 raft_consensus.cc:359] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.102370 18223 raft_consensus.cc:385] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:49.102618 18223 raft_consensus.cc:740] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
W20260924 19:57:49.105419 18927 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:55157 (request call id 40) took 799 ms (client timeout 1000 ms). Trace:
W20260924 19:57:49.107071 18927 rpcz_store.cc:269] 0924 19:57:48.305366 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:48.319304 (+ 13938us) service_pool.cc:224] Handling call
0924 19:57:49.092033 (+772729us) spinlock_profiling.cc:243] Waited 758 ms on lock 0x7b54000604d4. stack: 00007fa957217323 00007fa957217220 00007fa956dbb574 0000000000516964 00000000005168f9 00007fa95e3155c0 00007fa95e3b46fa 00007fa96012498c 00007fa960222c1e 00007fa95ddc8fb4 00007fa95ddc8f42 00007fa95ddc8e71 00007fa95ddc8dec 00007fa95ddc80a2 00007fa95c07a09c 00007fa95c0794d6
0924 19:57:49.105133 (+ 13100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1742768000}
I20260924 19:57:49.121739 18224 tablet_bootstrap.cc:654] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:49.123250 19329 consensus_queue.cc:1048] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:49.128001 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78801b8d0b664217a74b20160ba2d8ac" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:49.128012 18223 consensus_queue.cc:260] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:49.129558 18050 leader_election.cc:343] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:49.136801 18220 tablet_bootstrap.cc:492] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:49.138625 19204 leader_election.cc:290] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:49.149850 19213 raft_consensus.cc:493] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:49.150395 19213 raft_consensus.cc:515] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:49.152552 19213 leader_election.cc:290] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:49.153483 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78801b8d0b664217a74b20160ba2d8ac" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:49.153779 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a1ac7e1b192465696640c02898bc39f" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:49.154035 17810 raft_consensus.cc:3060] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:49.154985 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_5aeda90fd8e3425a8c49a239533690c9 [id=63622937f5a64a05b56aa1310407ec99]
I20260924 19:57:49.157344 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_7c27cdce0a3b40f69142de52aaf39493 [id=65f397bcbc164d0caa6cecae1bab74da]
I20260924 19:57:49.160745 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a1ac7e1b192465696640c02898bc39f" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:49.161932 17966 raft_consensus.cc:3060] T 57008266b80345e0a3f8e60bd7e2d62b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:49.162621 17753 leader_election.cc:343] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:49.166234 17753 leader_election.cc:343] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:49.166604 17753 leader_election.cc:304] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:49.167309 19213 raft_consensus.cc:2749] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:49.171530 19328 consensus_queue.cc:1048] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:49.138062 18220 ts_tablet_manager.cc:1403] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.225s user 0.009s sys 0.002s
I20260924 19:57:49.180146 17810 raft_consensus.cc:2468] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:49.181387 18051 leader_election.cc:304] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:49.182066 18220 raft_consensus.cc:359] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:49.182684 18220 raft_consensus.cc:385] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:57:49.086078 17600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:57:49.097759 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:49.188436 19080 raft_consensus.cc:2804] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:49.189029 19080 raft_consensus.cc:697] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
W20260924 19:57:49.134443 17601 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 78801b8d0b664217a74b20160ba2d8ac (table default.Table_454683a3d79d4988b41292f7776df549 [id=81b26242949442c0a7db6f55f0f18097], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 78801b8d0b664217a74b20160ba2d8ac consensus state has no leader
I20260924 19:57:49.190013 19080 consensus_queue.cc:237] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.170552 17608 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 273) took 2438 ms. Trace:
I20260924 19:57:49.203318 19213 raft_consensus.cc:493] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:49.202564 17608 rpcz_store.cc:276] 0924 19:57:46.732377 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.169554 (+2437177us) service_pool.cc:224] Handling call
0924 19:57:49.170473 (+ 919us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:49.204119 19213 raft_consensus.cc:515] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.206393 19213 leader_election.cc:290] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:49.210953 18175 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.2.188:38761)
I20260924 19:57:49.215534 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ba365c977984e68a316356465b1e36a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:49.216241 18117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ba365c977984e68a316356465b1e36a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:49.219290 17753 leader_election.cc:343] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:49.219954 17753 leader_election.cc:343] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:49.220347 17753 leader_election.cc:304] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:49.221369 19329 consensus_queue.cc:1048] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:49.231313 19341 raft_consensus.cc:2749] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:49.182960 18220 raft_consensus.cc:740] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:49.234833 18220 consensus_queue.cc:260] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:49.202805 17615 catalog_manager.cc:5938] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465) does not have the latest schema for tablet d129e0979a584f40ba9a975349cf15a5 (table default.Table_a12f8684de2041ec83f2fee77829afdd [id=94cea8de30ca4010a95f142ac8408653]). Expected version 1 got 0
W20260924 19:57:49.250644 17612 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet df722b0bbfd04ce2943f0e5b8058c50f consensus state has no leader
I20260924 19:57:49.251225 17612 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 257) took 3909 ms. Trace:
I20260924 19:57:49.251520 17612 rpcz_store.cc:276] 0924 19:57:45.341519 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.855174 (+513655us) service_pool.cc:224] Handling call
0924 19:57:45.941390 (+ 86216us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:45.957093 (+ 15703us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:45.957740 (+ 647us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:45.957962 (+ 222us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:46.062825 (+104863us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:49.251139 (+3188314us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:45.987263 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:46.009386 (+ 22123us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:46.009418 (+ 32us) write_op.cc:435] Acquired schema lock
0924 19:57:46.009435 (+ 17us) tablet.cc:662] Decoding operations
0924 19:57:46.009717 (+ 282us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:46.009760 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:46.009788 (+ 28us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:46.009944 (+ 156us) tablet.cc:701] Row locks acquired
0924 19:57:46.009960 (+ 16us) write_op.cc:260] PREPARE: finished
0924 19:57:46.010093 (+ 133us) write_op.cc:270] Start()
0924 19:57:46.010179 (+ 86us) write_op.cc:276] Timestamp: P: 1790279866010044 usec, L: 0
0924 19:57:46.010196 (+ 17us) op_driver.cc:348] REPLICATION: starting
0924 19:57:46.010710 (+ 514us) log.cc:844] Serialized 405 byte log entry
0924 19:57:46.057643 (+ 46933us) op_driver.cc:464] REPLICATION: finished
0924 19:57:46.058011 (+ 368us) write_op.cc:301] APPLY: starting
0924 19:57:46.058081 (+ 70us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:46.058182 (+ 101us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:46.058289 (+ 107us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:46.058468 (+ 179us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:46.058603 (+ 135us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:46.058646 (+ 43us) write_op.cc:312] APPLY: finished
0924 19:57:46.058825 (+ 179us) log.cc:844] Serialized 23 byte log entry
0924 19:57:46.058993 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:46.059057 (+ 64us) write_op.cc:454] Released schema lock
0924 19:57:46.059094 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:46.059845 (+ 751us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:46.059872 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":203,"apply.run_cpu_time_us":1317,"apply.run_wall_time_us":1798,"num_ops":1,"prepare.queue_time_us":15305,"prepare.run_cpu_time_us":1993,"prepare.run_wall_time_us":23631,"replication_time_us":47335}]]}
I20260924 19:57:49.255373 17606 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 97a1b8ccd35a43cea7a71d705fad1494
I20260924 19:57:49.251535 17613 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:51280 (request call id 255) took 3955 ms. Trace:
I20260924 19:57:49.257642 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 97a1b8ccd35a43cea7a71d705fad1494 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:46 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:49.258068 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 97a1b8ccd35a43cea7a71d705fad1494 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:46 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:49.256870 17606 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 270) took 2669 ms. Trace:
I20260924 19:57:49.260062 17606 rpcz_store.cc:276] 0924 19:57:46.587663 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:46.587757 (+ 94us) service_pool.cc:224] Handling call
0924 19:57:46.589288 (+ 1531us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:46.590314 (+ 1026us) catalog_manager.cc:2854] Modifying in-memory table state
0924 19:57:46.590731 (+ 417us) catalog_manager.cc:2863] Locking tablets
0924 19:57:46.708534 (+117803us) catalog_manager.cc:2877] Removing table and tablets from system table
0924 19:57:46.977737 (+269203us) catalog_manager.cc:2895] Removing table from by-name map
0924 19:57:46.977942 (+ 205us) catalog_manager.cc:2911] Committing in-memory state
0924 19:57:49.255287 (+2277345us) catalog_manager.cc:2915] Aborting table tasks
0924 19:57:49.256815 (+ 1528us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:46.732793 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:46.733309 (+ 516us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:46.733334 (+ 25us) write_op.cc:435] Acquired schema lock
0924 19:57:46.733347 (+ 13us) tablet.cc:662] Decoding operations
0924 19:57:46.733667 (+ 320us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:46.733703 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:46.733724 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:57:46.734023 (+ 299us) tablet.cc:701] Row locks acquired
0924 19:57:46.734037 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:57:46.734143 (+ 106us) write_op.cc:270] Start()
0924 19:57:46.734223 (+ 80us) write_op.cc:276] Timestamp: P: 1790279866734077 usec, L: 0
0924 19:57:46.734238 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:57:46.789614 (+ 55376us) log.cc:844] Serialized 768 byte log entry
0924 19:57:46.927776 (+138162us) op_driver.cc:464] REPLICATION: finished
0924 19:57:46.975754 (+ 47978us) write_op.cc:301] APPLY: starting
0924 19:57:46.975817 (+ 63us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:46.975926 (+ 109us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:46.976023 (+ 97us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:46.976321 (+ 298us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:46.976461 (+ 140us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0924 19:57:46.976503 (+ 42us) write_op.cc:312] APPLY: finished
0924 19:57:46.976684 (+ 181us) log.cc:844] Serialized 29 byte log entry
0924 19:57:46.976835 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:46.976899 (+ 64us) write_op.cc:454] Released schema lock
0924 19:57:46.976931 (+ 32us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47812,"apply.run_cpu_time_us":1364,"apply.run_wall_time_us":1365,"num_ops":2,"prepare.queue_time_us":22420,"prepare.run_cpu_time_us":2388,"prepare.run_wall_time_us":58181,"raft.queue_time_us":537,"raft.run_cpu_time_us":580,"raft.run_wall_time_us":580,"replication_time_us":193440}]]}
I20260924 19:57:49.257189 17613 rpcz_store.cc:276] 0924 19:57:45.296010 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.379010 (+ 83000us) service_pool.cc:224] Handling call
0924 19:57:45.525664 (+146654us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa5522077e6 00007fa54fdbc634 00007fa54fdbc5c2 00007fa54fdbc4f1 00007fa54fdbc46c 00007fa54fdbb722 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:45.526314 (+ 650us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:57:45.526997 (+ 683us) catalog_manager.cc:3764] Apply alter schema
0924 19:57:45.527552 (+ 555us) catalog_manager.cc:3796] Acquired catalog manager lock
0924 19:57:45.527730 (+ 178us) catalog_manager.cc:3954] Updating metadata on disk
0924 19:57:45.655418 (+127688us) catalog_manager.cc:3985] Committing alterations to in-memory state
0924 19:57:49.251450 (+3596032us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:45.589681 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:45.590570 (+ 889us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:45.590634 (+ 64us) write_op.cc:435] Acquired schema lock
0924 19:57:45.590647 (+ 13us) tablet.cc:662] Decoding operations
0924 19:57:45.629001 (+ 38354us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:45.629151 (+ 150us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:45.629286 (+ 135us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:45.629918 (+ 632us) tablet.cc:701] Row locks acquired
0924 19:57:45.629933 (+ 15us) write_op.cc:260] PREPARE: finished
0924 19:57:45.630362 (+ 429us) write_op.cc:270] Start()
0924 19:57:45.630707 (+ 345us) write_op.cc:276] Timestamp: P: 1790279865630194 usec, L: 0
0924 19:57:45.630725 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:57:45.632658 (+ 1933us) log.cc:844] Serialized 405 byte log entry
0924 19:57:45.647957 (+ 15299us) op_driver.cc:464] REPLICATION: finished
0924 19:57:45.653593 (+ 5636us) write_op.cc:301] APPLY: starting
0924 19:57:45.653655 (+ 62us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:45.653757 (+ 102us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:45.653848 (+ 91us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:45.654040 (+ 192us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:45.654169 (+ 129us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:45.654219 (+ 50us) write_op.cc:312] APPLY: finished
0924 19:57:45.654425 (+ 206us) log.cc:844] Serialized 23 byte log entry
0924 19:57:45.655081 (+ 656us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.655149 (+ 68us) write_op.cc:454] Released schema lock
0924 19:57:45.655206 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:45.655449 (+ 243us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:45.655476 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":335551872,"child_traces":[["op",{"apply.queue_time_us":5466,"apply.run_cpu_time_us":1350,"apply.run_wall_time_us":1831,"num_ops":1,"prepare.queue_time_us":59713,"prepare.run_cpu_time_us":3347,"prepare.run_wall_time_us":44954,"raft.queue_time_us":3331,"raft.run_cpu_time_us":761,"raft.run_wall_time_us":6005,"replication_time_us":17087}]]}
I20260924 19:57:49.266126 17613 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 275) took 2452 ms. Trace:
I20260924 19:57:49.267850 17613 rpcz_store.cc:276] 0924 19:57:46.813394 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.265169 (+2451775us) service_pool.cc:224] Handling call
0924 19:57:49.266060 (+ 891us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:49.269297 17613 master_service.cc:960] ReplaceTablet: received request to replace tablet accddc7f696142bfa6a5e655f5abc220 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:49.270330 17613 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51280 (request call id 276) took 2453 ms. Trace:
I20260924 19:57:49.276340 19336 raft_consensus.cc:493] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:49.276800 19336 raft_consensus.cc:515] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:49.278779 19336 leader_election.cc:290] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:49.281553 18220 ts_tablet_manager.cc:1434] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.102s user 0.006s sys 0.000s
I20260924 19:57:49.282529 18220 tablet_bootstrap.cc:492] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:49.287258 17613 rpcz_store.cc:276] 0924 19:57:46.816459 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.269196 (+2452737us) service_pool.cc:224] Handling call
0924 19:57:49.270277 (+ 1081us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:49.288621 17613 master_service.cc:960] ReplaceTablet: received request to replace tablet 484517ee7df8463bb70d201ead1c6835 from {username='slave'} at 127.0.0.1:51280
I20260924 19:57:49.289698 17613 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:51280 (request call id 279) took 1514 ms. Trace:
I20260924 19:57:49.290014 17613 rpcz_store.cc:276] 0924 19:57:47.775221 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.288514 (+1513293us) service_pool.cc:224] Handling call
0924 19:57:49.289643 (+ 1129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:49.276345 17606 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 277) took 2275 ms. Trace:
I20260924 19:57:49.290858 17606 rpcz_store.cc:276] 0924 19:57:47.000692 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.275513 (+2274821us) service_pool.cc:224] Handling call
0924 19:57:49.276287 (+ 774us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:49.294251 17606 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:53508 (request call id 1) took 1902 ms. Trace:
I20260924 19:57:49.294559 17606 rpcz_store.cc:276] 0924 19:57:47.391433 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.293411 (+1901978us) service_pool.cc:224] Handling call
0924 19:57:49.294196 (+ 785us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:49.278152 17608 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:51280 (request call id 274) took 2522 ms. Trace:
I20260924 19:57:49.296465 17608 rpcz_store.cc:276] 0924 19:57:46.755889 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.204975 (+2449086us) service_pool.cc:224] Handling call
0924 19:57:49.278098 (+ 73123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:49.304490 18223 ts_tablet_manager.cc:1434] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.246s user 0.008s sys 0.000s
W20260924 19:57:49.306150 17609 catalog_manager.cc:6384] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: An error occurred while updating sys-tables: Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER
W20260924 19:57:49.280827 18903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.188:38761, user_credentials={real_user=slave}} blocked reactor thread for 88700.9us
W20260924 19:57:49.329434 17607 catalog_manager.cc:6384] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: An error occurred while updating sys-tables: Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER
I20260924 19:57:49.312819 17612 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:51280 (request call id 278) took 1544 ms. Trace:
I20260924 19:57:49.335348 17612 rpcz_store.cc:276] 0924 19:57:47.738995 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.282303 (+1543308us) service_pool.cc:224] Handling call
0924 19:57:49.283108 (+ 805us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:49.337401 18320 master_runner.cc:233] Existing masters have differing terms: 3 vs 4
I20260924 19:57:49.338011 18320 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260924 19:57:49.339883 17614 catalog_manager.cc:6384] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: An error occurred while updating sys-tables: Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER
I20260924 19:57:49.343731 18082 tablet_service.cc:1558] Processing DeleteTablet for tablet 44199d2655f14dcf97afe2180789aae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:49.315757 18091 tablet_service.cc:1558] Processing DeleteTablet for tablet 97a1b8ccd35a43cea7a71d705fad1494 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:46 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:49.346777 19369 tablet_replica.cc:333] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
W20260924 19:57:49.351716 17611 catalog_manager.cc:6384] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: An error occurred while updating sys-tables: Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER
I20260924 19:57:49.356791 19369 raft_consensus.cc:2243] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:49.365346 19369 raft_consensus.cc:2272] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:49.370664 19369 ts_tablet_manager.cc:1916] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:49.377920 17609 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.130:37869 (request call id 44) took 3635 ms (client timeout 1000 ms). Trace:
I20260924 19:57:49.385072 18223 tablet_bootstrap.cc:492] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:49.386271 17609 rpcz_store.cc:269] 0924 19:57:45.742873 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.852251 (+109378us) service_pool.cc:224] Handling call
0924 19:57:49.305921 (+3453670us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.305943 (+ 22us) write_op.cc:454] Released schema lock
0924 19:57:49.377849 (+ 71906us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:49.305702 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER)
0924 19:57:49.305740 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.305760 (+ 20us) write_op.cc:454] Released schema lock
0924 19:57:49.305792 (+ 32us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":7152,"reported_tablets":26,"spinlock_wait_cycles":896,"child_traces":[["op",{}]]}
W20260924 19:57:49.390300 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 80049us
I20260924 19:57:49.393453 18091 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c861e93ad9f40ba9ad010021e43bc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:43 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:49.408829 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_ca51e47e4fc44d9ba5dd07cbeefa6e68 [id=6ea0f9084fd84e848fdecdd696cdc773]
I20260924 19:57:49.410279 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_388c160758d44e1e89b8cb5d541316f8 [id=70f30a75b8d443a0ae28243adf40862a]
W20260924 19:57:49.412221 17521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.188:38761, user_credentials={real_user=slave}} blocked reactor thread for 158376us
I20260924 19:57:49.423256 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c861e93ad9f40ba9ad010021e43bc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:43 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:49.433027 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_88b623b6a00845afb22e24e34604e9ea [id=72c7d8cc98ee49679d8c7589a9ac0c8f]
I20260924 19:57:49.439630 18224 tablet_bootstrap.cc:492] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:49.440078 18224 ts_tablet_manager.cc:1403] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.387s user 0.011s sys 0.005s
I20260924 19:57:49.449720 19377 tablet_replica.cc:333] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
W20260924 19:57:49.448750 17614 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 34) took 3961 ms (client timeout 1000 ms). Trace:
W20260924 19:57:49.461149 17599 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.AlterSchema -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 55638.5us
W20260924 19:57:49.462688 17615 catalog_manager.cc:6384] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: An error occurred while updating sys-tables: Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER
I20260924 19:57:49.463323 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 1d95d6e48173436c8c59130464413ea1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:46 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:49.464080 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 201ea0020cb3430eac85e80b8c06fb07 with delete_type TABLET_DATA_DELETED (replaced by tablet a00e3626ea154dc5a959f0fbb4e849dc) from {username='slave'} at 127.0.0.1:36562
W20260924 19:57:49.461943 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 44199d2655f14dcf97afe2180789aae4 because the tablet was not found. No further retry: Not found: Tablet not found: 44199d2655f14dcf97afe2180789aae4
I20260924 19:57:49.467202 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet ae7a0660795e4e4b87d0959ca84d5d1b with delete_type TABLET_DATA_DELETED (replaced by tablet bea1ebdf647e4ef9a131a0398672b8ab) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:49.468470 18095 tablet_service.cc:1558] Processing DeleteTablet for tablet 201ea0020cb3430eac85e80b8c06fb07 with delete_type TABLET_DATA_DELETED (replaced by tablet a00e3626ea154dc5a959f0fbb4e849dc) from {username='slave'} at 127.0.0.1:38450
I20260924 19:57:49.469182 18224 raft_consensus.cc:359] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.469825 18224 raft_consensus.cc:385] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:49.470065 18224 raft_consensus.cc:740] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:49.473894 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet ba46ac86b242407abd3e8dbf09f12efa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:49.475875 17796 tablet_service.cc:1558] Processing DeleteTablet for tablet 3192f66a492442c1bf4137cfe7e13bc9 with delete_type TABLET_DATA_DELETED (replaced by tablet 144355b18c2f413baa28dd632623db52) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:49.476527 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 44199d2655f14dcf97afe2180789aae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:42 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:49.477183 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet ae7a0660795e4e4b87d0959ca84d5d1b with delete_type TABLET_DATA_DELETED (replaced by tablet bea1ebdf647e4ef9a131a0398672b8ab) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:49.479808 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 07f86f24d9d14d05898e1c0f69428850 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d95d6e48173436c8c59130464413ea1) from {username='slave'} at 127.0.0.1:38450
W20260924 19:57:49.490054 17607 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 46) took 4959 ms (client timeout 1000 ms). Trace:
I20260924 19:57:49.349162 19370 tablet_replica.cc:333] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:57:49.491916 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 8aef0359a77240679240823612507ee8 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b0b011357674a42a09d83f9384abaf1) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:49.492354 19263 consensus_queue.cc:1048] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:49.493063 19370 raft_consensus.cc:2243] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:57:49.494189 19370 raft_consensus.cc:2272] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:49.435583 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]
I20260924 19:57:49.496294 19370 ts_tablet_manager.cc:1916] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:49.497895 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_a82c256e36e448c296c86a30176f4135 [id=757ccfb2d39f4907a2580b365d283ace]
I20260924 19:57:49.504693 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 57008266b80345e0a3f8e60bd7e2d62b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:37 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:49.451421 19377 raft_consensus.cc:2243] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:49.508913 19377 raft_consensus.cc:2272] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:49.517318 18224 consensus_queue.cc:260] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.519409 18224 ts_tablet_manager.cc:1434] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.079s user 0.006s sys 0.001s
I20260924 19:57:49.521991 19263 raft_consensus.cc:493] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:49.308439 17906 leader_election.cc:343] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 1d95d6e48173436c8c59130464413ea1
I20260924 19:57:49.522481 19263 raft_consensus.cc:515] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:49.523244 17905 leader_election.cc:343] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 1d95d6e48173436c8c59130464413ea1
I20260924 19:57:49.525434 19263 leader_election.cc:290] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:49.525381 17905 leader_election.cc:304] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:49.526160 19336 raft_consensus.cc:2749] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:49.526845 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 209162us
I20260924 19:57:49.528867 19374 raft_consensus.cc:493] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:49.529270 19374 raft_consensus.cc:515] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.531642 19374 leader_election.cc:290] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:49.540045 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 231645us
I20260924 19:57:49.541122 18220 tablet_bootstrap.cc:700] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a: Time spent opening tablet: real 0.254s user 0.004s sys 0.000s
I20260924 19:57:49.555794 18220 tablet_bootstrap.cc:654] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:49.566085 18223 tablet_bootstrap.cc:654] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:49.567456 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:33 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:49.568477 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 9c861e93ad9f40ba9ad010021e43bc2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:43 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:49.569319 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 07f86f24d9d14d05898e1c0f69428850 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d95d6e48173436c8c59130464413ea1) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:49.570153 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 201ea0020cb3430eac85e80b8c06fb07 with delete_type TABLET_DATA_DELETED (replaced by tablet a00e3626ea154dc5a959f0fbb4e849dc) from {username='slave'} at 127.0.0.1:51332
I20260924 19:57:49.570879 19336 raft_consensus.cc:493] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:49.511099 19377 ts_tablet_manager.cc:1916] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:49.571403 19336 raft_consensus.cc:515] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.573230 19336 leader_election.cc:290] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:49.574966 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96379e7356ab40218c6ee2c126c0b4ae" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:49.576828 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9e7c5b28fea4dbda802702c8b85ec54" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:49.579784 19336 consensus_queue.cc:1048] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:49.580118 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_0c6b7fd4e7bd42a3ba0d407aea438273 [id=79e6d48b52b740958363f06cf05f2f63]
W20260924 19:57:49.580633 17615 catalog_manager.cc:6384] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: An error occurred while updating sys-tables: Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER
I20260924 19:57:49.581674 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_a10031a0fa004ac3ad48e6020477494b [id=7aab4014a5bc4823bbe845c021153fc9]
I20260924 19:57:49.584442 17823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96379e7356ab40218c6ee2c126c0b4ae" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:49.584982 17823 raft_consensus.cc:2468] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
W20260924 19:57:49.591218 18899 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.189:43689 timed out after 0.500s (SENT)
I20260924 19:57:49.606715 18051 leader_election.cc:304] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
W20260924 19:57:49.612139 18050 leader_election.cc:343] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:49.612028 19389 tablet.cc:1751] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260924 19:57:49.613400 18050 leader_election.cc:343] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:49.625824 19204 raft_consensus.cc:2804] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:49.626185 19204 raft_consensus.cc:493] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:49.626493 19204 raft_consensus.cc:3060] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:49.632675 18220 tablet_bootstrap.cc:492] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:49.633190 18220 ts_tablet_manager.cc:1403] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.351s user 0.009s sys 0.003s
I20260924 19:57:49.638123 18224 tablet_bootstrap.cc:492] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:49.642304 19204 raft_consensus.cc:515] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:49.452167 17614 rpcz_store.cc:269] 0924 19:57:45.487213 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.491747 (+ 4534us) service_pool.cc:224] Handling call
0924 19:57:45.637783 (+146036us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b5400030554. stack: 00007fa5491fa323 00007fa5491fa220 00007fa548d9e574 0000000000516964 00000000005168f9 00007fa5502f85c0 00007fa5503976fa 00007fa55210798c 00007fa552205c1e 00007fa54fdabfb4 00007fa54fdabf42 00007fa54fdabe71 00007fa54fdabdec 00007fa54fdab0a2 00007fa54e05d09c 00007fa54e05c4d6
0924 19:57:49.339685 (+3701902us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.339705 (+ 20us) write_op.cc:454] Released schema lock
0924 19:57:49.448190 (+108485us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:46.453952 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:57:46.454336 (+ 384us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:57:46.454360 (+ 24us) write_op.cc:435] Acquired schema lock
0924 19:57:46.454390 (+ 30us) tablet.cc:662] Decoding operations
0924 19:57:46.454631 (+ 241us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:57:46.454669 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0924 19:57:46.454688 (+ 19us) tablet.cc:685] Acquiring locks for 1 operations
0924 19:57:46.454826 (+ 138us) tablet.cc:701] Row locks acquired
0924 19:57:46.454837 (+ 11us) write_op.cc:260] PREPARE: finished
0924 19:57:46.454948 (+ 111us) write_op.cc:270] Start()
0924 19:57:46.455022 (+ 74us) write_op.cc:276] Timestamp: P: 1790279866454908 usec, L: 0
0924 19:57:46.455037 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:57:46.455557 (+ 520us) log.cc:844] Serialized 530 byte log entry
0924 19:57:46.474812 (+ 19255us) op_driver.cc:464] REPLICATION: finished
0924 19:57:46.475175 (+ 363us) write_op.cc:301] APPLY: starting
0924 19:57:46.475237 (+ 62us) tablet.cc:1394] starting BulkCheckPresence
0924 19:57:46.475347 (+ 110us) tablet.cc:1397] finished BulkCheckPresence
0924 19:57:46.475455 (+ 108us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:57:46.475637 (+ 182us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:57:46.475775 (+ 138us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0924 19:57:46.475820 (+ 45us) write_op.cc:312] APPLY: finished
0924 19:57:46.476012 (+ 192us) log.cc:844] Serialized 23 byte log entry
0924 19:57:46.476162 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:46.476225 (+ 63us) write_op.cc:454] Released schema lock
0924 19:57:46.476261 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0924 19:57:46.476487 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:46.476512 (+ 25us) write_op.cc:454] Released schema lock
Related trace 'op':
0924 19:57:49.339489 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER)
0924 19:57:49.339538 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.339569 (+ 31us) write_op.cc:454] Released schema lock
0924 19:57:49.339586 (+ 17us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":12513,"reported_tablets":34,"spinlock_wait_cycles":317019520,"child_traces":[["op",{"apply.queue_time_us":193,"apply.run_cpu_time_us":1301,"apply.run_wall_time_us":1301,"num_ops":1,"prepare.queue_time_us":564,"prepare.run_cpu_time_us":1869,"prepare.run_wall_time_us":1869,"raft.queue_time_us":1043,"raft.run_cpu_time_us":663,"raft.run_wall_time_us":663,"replication_time_us":19671}],["op",{}]]}
W20260924 19:57:49.490410 17607 rpcz_store.cc:269] 0924 19:57:44.530525 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.094476 (+563951us) service_pool.cc:224] Handling call
0924 19:57:49.329226 (+4234750us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.329246 (+ 20us) write_op.cc:454] Released schema lock
0924 19:57:49.489982 (+160736us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:49.329064 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER)
0924 19:57:49.329096 (+ 32us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.329115 (+ 19us) write_op.cc:454] Released schema lock
0924 19:57:49.329133 (+ 18us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":182166,"reported_tablets":38,"child_traces":[["op",{}]]}
W20260924 19:57:49.602839 17600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:57:49.647482 17599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.189:43689 timed out after 0.582s (SENT) [suppressed 1 similar messages]
W20260924 19:57:49.657848 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 45052.9us
W20260924 19:57:49.658694 17906 consensus_peers.cc:597] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3192f66a492442c1bf4137cfe7e13bc9. This is attempt 1: this message will repeat every 5th retry.
W20260924 19:57:49.659399 17906 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:57:49.660635 19204 leader_election.cc:290] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:49.664064 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b [id=7babe5418e104e76871835ca3e30f95c]
I20260924 19:57:49.666059 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_5208198cb618443dbec0f760012e4edd [id=7cb855f3577d4824ac7edb72b8f7072c]
I20260924 19:57:49.676420 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9e7c5b28fea4dbda802702c8b85ec54" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:49.677055 17816 raft_consensus.cc:2468] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:49.688424 18223 tablet_bootstrap.cc:492] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:49.688865 18223 ts_tablet_manager.cc:1403] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.304s user 0.007s sys 0.003s
I20260924 19:57:49.689397 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96379e7356ab40218c6ee2c126c0b4ae" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:49.689973 17818 raft_consensus.cc:3060] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:49.703480 18223 raft_consensus.cc:359] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.704111 18223 raft_consensus.cc:385] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:49.704348 18223 raft_consensus.cc:740] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:49.704878 18223 consensus_queue.cc:260] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.712533 19363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:49.713238 19363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: This master's current role is: LEADER
W20260924 19:57:49.723059 17599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.189:43689, user_credentials={real_user=slave}} blocked reactor thread for 75589.6us
I20260924 19:57:49.707760 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_aeac9df2b26b490a83f10d672335d75e [id=80222cdc26da492e9f386e9eb2fb77b8]
W20260924 19:57:49.616580 17611 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 47) took 3985 ms (client timeout 1000 ms). Trace:
W20260924 19:57:49.744330 17611 rpcz_store.cc:269] 0924 19:57:45.631359 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:45.847082 (+215723us) service_pool.cc:224] Handling call
0924 19:57:49.351515 (+3504433us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.351536 (+ 21us) write_op.cc:454] Released schema lock
0924 19:57:49.616488 (+264952us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:49.351331 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER)
0924 19:57:49.351365 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.351385 (+ 20us) write_op.cc:454] Released schema lock
0924 19:57:49.351415 (+ 30us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":4575,"reported_tablets":38,"child_traces":[["op",{}]]}
I20260924 19:57:49.761519 19370 ts_tablet_manager.cc:1929] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:49.762033 19370 log.cc:1199] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/97a1b8ccd35a43cea7a71d705fad1494
I20260924 19:57:49.762977 19370 ts_tablet_manager.cc:1950] T 97a1b8ccd35a43cea7a71d705fad1494 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:57:49.766815 18220 raft_consensus.cc:359] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.773463 18220 raft_consensus.cc:385] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:49.773837 18220 raft_consensus.cc:740] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:49.775272 18220 consensus_queue.cc:260] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.792807 17818 raft_consensus.cc:2468] T 96379e7356ab40218c6ee2c126c0b4ae P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:49.798445 19369 ts_tablet_manager.cc:1929] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:49.798928 19369 log.cc:1199] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/97a1b8ccd35a43cea7a71d705fad1494
I20260924 19:57:49.799757 19369 ts_tablet_manager.cc:1950] T 97a1b8ccd35a43cea7a71d705fad1494 P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:49.805259 18224 tablet_bootstrap.cc:654] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:49.811026 19336 raft_consensus.cc:493] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:49.811462 19336 raft_consensus.cc:515] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:49.813217 19336 leader_election.cc:290] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:49.814647 18220 ts_tablet_manager.cc:1434] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.181s user 0.007s sys 0.000s
I20260924 19:57:49.816223 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_454683a3d79d4988b41292f7776df549 [id=81b26242949442c0a7db6f55f0f18097]
I20260924 19:57:49.821532 19213 raft_consensus.cc:493] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:49.822029 19213 raft_consensus.cc:515] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.824028 19213 leader_election.cc:290] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:49.828087 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c058a49bfd5544b58b76edfd3fadbd1a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:49.829571 17753 leader_election.cc:343] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:49.835089 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 175077us
I20260924 19:57:49.839291 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_ae1050d7716043fbad49a8f62505e719 [id=8681c352f02242a69511794ee585bf28]
I20260924 19:57:49.853191 18224 tablet_bootstrap.cc:492] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:49.869973 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 97a1b8ccd35a43cea7a71d705fad1494 (table default.Table_850e5158d78d4850b87445dac75ef7a9 [id=f768ce157a7b407d9f1e55a5b9d0720a]) successfully deleted
I20260924 19:57:49.887591 19336 consensus_queue.cc:1048] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:49.888794 17906 consensus_peers.cc:597] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae7a0660795e4e4b87d0959ca84d5d1b. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:49.853610 18224 ts_tablet_manager.cc:1403] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.216s user 0.008s sys 0.003s
I20260924 19:57:49.901516 18223 ts_tablet_manager.cc:1434] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.212s user 0.004s sys 0.004s
I20260924 19:57:49.857257 19401 tablet.cc:1751] T 92c0daa7d67e4aa390e334c669adda76 P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:49.881814 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96379e7356ab40218c6ee2c126c0b4ae" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:49.837505 18220 tablet_bootstrap.cc:492] T a9e7c5b28fea4dbda802702c8b85ec54 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:49.907105 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet 97a1b8ccd35a43cea7a71d705fad1494 (table default.Table_850e5158d78d4850b87445dac75ef7a9 [id=f768ce157a7b407d9f1e55a5b9d0720a]) successfully deleted
I20260924 19:57:49.886317 19395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:49.916795 19401 tablet.cc:1751] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:49.917477 17809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1137318a9fa4a0781f718dc34c4ecbe" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
W20260924 19:57:49.905246 17905 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:57:49.918252 17809 raft_consensus.cc:2393] T e1137318a9fa4a0781f718dc34c4ecbe P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:49.918277 17905 leader_election.cc:304] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:49.920102 19336 raft_consensus.cc:2749] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:49.916882 19395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:49.923604 19390 tablet.cc:1751] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260924 19:57:49.930570 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet fec9b6ecfb384b528cf71c4cfb2fd4cb because the tablet was not found. No further retry: Not found: Tablet not found: fec9b6ecfb384b528cf71c4cfb2fd4cb
W20260924 19:57:49.931183 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 9c861e93ad9f40ba9ad010021e43bc2b because the tablet was not found. No further retry: Not found: Tablet not found: 9c861e93ad9f40ba9ad010021e43bc2b
W20260924 19:57:49.931718 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 07f86f24d9d14d05898e1c0f69428850 because the tablet was not found. No further retry: Not found: Tablet not found: 07f86f24d9d14d05898e1c0f69428850
W20260924 19:57:49.932217 17599 catalog_manager.cc:5030] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): delete failed for tablet 201ea0020cb3430eac85e80b8c06fb07 because the tablet was not found. No further retry: Not found: Tablet not found: 201ea0020cb3430eac85e80b8c06fb07
I20260924 19:57:49.932226 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet 97a1b8ccd35a43cea7a71d705fad1494 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:46 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:57:49.912207 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c058a49bfd5544b58b76edfd3fadbd1a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:49.953735 17615 catalog_manager.cc:6038] Failed to send AlterTable RPC for tablet e1137318a9fa4a0781f718dc34c4ecbe (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e1137318a9fa4a0781f718dc34c4ecbe consensus state has no leader
I20260924 19:57:49.959004 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_e3179b11ea114ce99779ed0b2ac46732 [id=8c39c8f34038451d840db3c909d76f3d]
I20260924 19:57:49.960580 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_2a8be69ab9604081a07df5197e7fab9d [id=8d382cdb38764b11ba323546fff603bf]
I20260924 19:57:49.964303 18220 tablet_bootstrap.cc:654] T a9e7c5b28fea4dbda802702c8b85ec54 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:49.964532 19398 consensus_queue.cc:1048] T ca37ddd22f5d4bcab2862553408447ce P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:49.968416 18224 raft_consensus.cc:359] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:49.969077 18224 raft_consensus.cc:385] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:49.969336 18224 raft_consensus.cc:740] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:49.969970 18224 consensus_queue.cc:260] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:49.986891 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 64112.2us
I20260924 19:57:50.009120 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]
I20260924 19:57:50.010792 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_7b49c63ad79c4132afbbbb9e1f2d4ca0 [id=8edea4510fd84d40819b48b401ccfe64]
I20260924 19:57:49.991884 19376 consensus_queue.cc:1048] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.058s
I20260924 19:57:50.023344 18224 ts_tablet_manager.cc:1434] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.132s user 0.006s sys 0.001s
I20260924 19:57:50.024312 18220 tablet_bootstrap.cc:492] T a9e7c5b28fea4dbda802702c8b85ec54 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:50.024714 18220 ts_tablet_manager.cc:1403] T a9e7c5b28fea4dbda802702c8b85ec54 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.187s user 0.008s sys 0.002s
I20260924 19:57:50.027062 18220 raft_consensus.cc:359] T a9e7c5b28fea4dbda802702c8b85ec54 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.027704 18220 raft_consensus.cc:385] T a9e7c5b28fea4dbda802702c8b85ec54 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:57:50.027719 17615 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.17.2.129:36093 (request call id 48) took 3185 ms (client timeout 1000 ms). Trace:
I20260924 19:57:50.027953 18220 raft_consensus.cc:740] T a9e7c5b28fea4dbda802702c8b85ec54 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:50.028652 18220 consensus_queue.cc:260] T a9e7c5b28fea4dbda802702c8b85ec54 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:50.028239 17615 rpcz_store.cc:269] 0924 19:57:46.842541 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:47.535838 (+693297us) service_pool.cc:224] Handling call
0924 19:57:49.462398 (+1926560us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.462431 (+ 33us) write_op.cc:454] Released schema lock
0924 19:57:49.580381 (+117950us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.580404 (+ 23us) write_op.cc:454] Released schema lock
0924 19:57:50.027565 (+447161us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:57:49.379045 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER)
0924 19:57:49.379080 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.379101 (+ 21us) write_op.cc:454] Released schema lock
0924 19:57:49.379121 (+ 20us) write_op.cc:335] FINISH: Op aborted
Related trace 'op':
0924 19:57:49.580178 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica e1ff67c5d57d474a9d92cc4c6de1287e is not leader of this config: current role FOLLOWER)
0924 19:57:49.580215 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:57:49.580237 (+ 22us) write_op.cc:454] Released schema lock
0924 19:57:49.580258 (+ 21us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":1252,"reported_tablets":41,"child_traces":[["op",{}],["op",{}]]}
W20260924 19:57:50.035003 18050 leader_election.cc:343] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:50.038564 17753 leader_election.cc:343] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:50.038941 17753 leader_election.cc:304] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:50.040328 17967 raft_consensus.cc:3060] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:50.040534 19213 raft_consensus.cc:2749] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:57:50.042335 17600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:57:50.058274 19263 consensus_queue.cc:1048] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:50.063910 18223 tablet_bootstrap.cc:492] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:50.067775 18220 ts_tablet_manager.cc:1434] T a9e7c5b28fea4dbda802702c8b85ec54 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20260924 19:57:50.068737 18220 tablet_bootstrap.cc:492] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
W20260924 19:57:50.083974 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 170136us
I20260924 19:57:50.084741 18051 leader_election.cc:304] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:50.085469 19204 raft_consensus.cc:2804] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:50.085806 19204 raft_consensus.cc:493] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:50.086094 19204 raft_consensus.cc:3060] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:50.092801 19263 consensus_queue.cc:1048] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:50.096856 18220 tablet_bootstrap.cc:654] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:50.103726 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 63588.3us
I20260924 19:57:50.113777 18051 leader_election.cc:304] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:50.114711 19374 raft_consensus.cc:2804] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:50.115173 19374 raft_consensus.cc:697] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:50.116003 19374 consensus_queue.cc:237] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:50.126595 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 9c861e93ad9f40ba9ad010021e43bc2b because the tablet was not found. No further retry: Not found: Tablet not found: 9c861e93ad9f40ba9ad010021e43bc2b
W20260924 19:57:50.128304 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 1d95d6e48173436c8c59130464413ea1 because the tablet was not found. No further retry: Not found: Tablet not found: 1d95d6e48173436c8c59130464413ea1
W20260924 19:57:50.128918 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 201ea0020cb3430eac85e80b8c06fb07 because the tablet was not found. No further retry: Not found: Tablet not found: 201ea0020cb3430eac85e80b8c06fb07
I20260924 19:57:50.129338 19369 tablet_replica.cc:333] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
W20260924 19:57:50.129482 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet ba46ac86b242407abd3e8dbf09f12efa because the tablet was not found. No further retry: Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
W20260924 19:57:50.130014 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 3192f66a492442c1bf4137cfe7e13bc9 because the tablet was not found. No further retry: Not found: Tablet not found: 3192f66a492442c1bf4137cfe7e13bc9
W20260924 19:57:50.130352 18899 leader_election.cc:336] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689): Timed out: RequestConsensusVote RPC to 127.17.2.189:43689 timed out after 0.500s (SENT)
I20260924 19:57:50.130775 18899 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
W20260924 19:57:50.131771 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 44199d2655f14dcf97afe2180789aae4 because the tablet was not found. No further retry: Not found: Tablet not found: 44199d2655f14dcf97afe2180789aae4
I20260924 19:57:50.132100 19395 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260924 19:57:50.132423 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet ae7a0660795e4e4b87d0959ca84d5d1b because the tablet was not found. No further retry: Not found: Tablet not found: ae7a0660795e4e4b87d0959ca84d5d1b
W20260924 19:57:50.132987 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 8aef0359a77240679240823612507ee8 because the tablet was not found. No further retry: Not found: Tablet not found: 8aef0359a77240679240823612507ee8
I20260924 19:57:50.133198 19369 raft_consensus.cc:2243] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:50.133816 19369 raft_consensus.cc:2272] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:50.136823 19369 ts_tablet_manager.cc:1916] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:57:50.135087 18224 tablet_bootstrap.cc:492] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:50.106518 19204 raft_consensus.cc:515] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.169991 19419 tablet.cc:1751] T 077f70bcee7e412299da12ddc30a63ff P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260924 19:57:50.171316 19421 tablet.cc:1751] T 92c0daa7d67e4aa390e334c669adda76 P c4f6e3a7b06148918d230c77ab0c8202: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260924 19:57:50.175601 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 54061.2us
I20260924 19:57:50.180193 19377 ts_tablet_manager.cc:1929] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:57:50.180606 19377 log.cc:1199] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/97a1b8ccd35a43cea7a71d705fad1494
I20260924 19:57:50.181361 19377 ts_tablet_manager.cc:1950] T 97a1b8ccd35a43cea7a71d705fad1494 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:50.181279 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9e7c5b28fea4dbda802702c8b85ec54" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:50.181978 17971 raft_consensus.cc:3060] T a9e7c5b28fea4dbda802702c8b85ec54 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:50.189420 17971 raft_consensus.cc:2468] T a9e7c5b28fea4dbda802702c8b85ec54 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:50.194276 18220 tablet_bootstrap.cc:492] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:50.194736 18220 ts_tablet_manager.cc:1403] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.126s user 0.011s sys 0.000s
I20260924 19:57:50.197011 18220 raft_consensus.cc:359] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.197623 18220 raft_consensus.cc:385] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:50.197885 18220 raft_consensus.cc:740] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:50.198493 18220 consensus_queue.cc:260] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.205260 19204 leader_election.cc:290] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:50.105429 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_a12f8684de2041ec83f2fee77829afdd [id=94cea8de30ca4010a95f142ac8408653]
I20260924 19:57:50.222064 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_b6d3fa192eb741158adf5b69323a29de [id=9ead6eed50694fd1b920b5b0d46891c9]
I20260924 19:57:50.236743 18223 tablet_bootstrap.cc:654] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:50.244530 18224 tablet_bootstrap.cc:654] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:50.247025 18220 ts_tablet_manager.cc:1434] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.052s user 0.006s sys 0.000s
I20260924 19:57:50.247872 18220 tablet_bootstrap.cc:492] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:50.253065 18220 tablet_bootstrap.cc:654] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:50.258383 18223 tablet_bootstrap.cc:492] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:50.258798 18223 ts_tablet_manager.cc:1403] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.195s user 0.006s sys 0.004s
I20260924 19:57:50.276783 19375 raft_consensus.cc:493] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:50.276849 18111 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1137318a9fa4a0781f718dc34c4ecbe" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:50.277347 19375 raft_consensus.cc:515] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:50.278491 17602 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet e1137318a9fa4a0781f718dc34c4ecbe (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e1137318a9fa4a0781f718dc34c4ecbe consensus state has no leader
I20260924 19:57:50.285799 18223 raft_consensus.cc:359] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:50.286481 18223 raft_consensus.cc:385] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:50.286732 18223 raft_consensus.cc:740] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
W20260924 19:57:50.287290 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 110371us
I20260924 19:57:50.303221 19377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 249) took 1004 ms. Trace:
I20260924 19:57:50.303679 19377 rpcz_store.cc:276] 0924 19:57:49.298978 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.299836 (+ 858us) service_pool.cc:224] Handling call
0924 19:57:49.449426 (+149590us) ts_tablet_manager.cc:1151] Deleting tablet 97a1b8ccd35a43cea7a71d705fad1494
0924 19:57:49.449503 (+ 77us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:49.720915 (+271412us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:50.180060 (+459145us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:50.303124 (+123064us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":119382,"tablet-delete.queue_time_us":99373}
I20260924 19:57:50.300784 18223 consensus_queue.cc:260] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:50.306118 19377 tablet_replica.cc:333] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:57:50.306495 18223 ts_tablet_manager.cc:1434] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.047s user 0.002s sys 0.004s
I20260924 19:57:50.307439 19377 raft_consensus.cc:2243] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:57:50.308609 19377 raft_consensus.cc:2272] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:57:50.310880 19377 ts_tablet_manager.cc:1916] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:57:50.312721 17599 leader_election.cc:336] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689): Timed out: RequestConsensusVote RPC to 127.17.2.189:43689 timed out after 0.582s (SENT)
I20260924 19:57:50.313156 17599 leader_election.cc:304] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1ff67c5d57d474a9d92cc4c6de1287e; no voters: 3e059bbe4dac4178bccb3be48cb1118d, 6f88e00b235f48bb82e6a3c9c9f51fa6
I20260924 19:57:50.321705 17821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9e7c5b28fea4dbda802702c8b85ec54" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:50.322315 17821 raft_consensus.cc:3060] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:50.330453 17821 raft_consensus.cc:2468] T a9e7c5b28fea4dbda802702c8b85ec54 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:50.307988 19375 leader_election.cc:290] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:50.320384 18223 tablet_bootstrap.cc:492] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:50.346141 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d930e7f4a6b7471899dab1b28e40f93d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:50.357802 17810 raft_consensus.cc:2468] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:50.358979 18223 tablet_bootstrap.cc:654] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:50.403335 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_0a22025767fb456485c2624d5ca0dcbb [id=9f151548ce074cb6b8b8497fa246d61a]
W20260924 19:57:50.418310 17599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.189:43689, user_credentials={real_user=slave}} blocked reactor thread for 105576us
I20260924 19:57:49.212852 17418 master-stress-test.cc:579] Killing test thread 1/10
W20260924 19:57:50.433246 18176 thread.cc:641] client-negotiator [worker] (thread pool) Time spent creating pthread: real 1.224s user 0.020s sys 0.039s
W20260924 19:57:50.433835 18176 thread.cc:608] client-negotiator [worker] (thread pool) Time spent starting thread: real 1.225s user 0.020s sys 0.039s
I20260924 19:57:50.450691 19429 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260924 19:57:49.296355 18175 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.17.2.188:38761)
W20260924 19:57:50.451525 18175 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.ReplaceTablet -> {remote=127.17.2.188:38761, user_credentials={real_user=slave}} blocked reactor thread for 1.15576e+06us
W20260924 19:57:50.452589 18175 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.17.2.188:38761)
W20260924 19:57:50.453351 18175 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.2.188:38761)
W20260924 19:57:50.455125 18175 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.17.2.188:38761)
I20260924 19:57:50.483537 19328 consensus_queue.cc:1048] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:50.487550 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 51364.6us
I20260924 19:57:50.489964 19328 consensus_queue.cc:1048] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:50.505405 17600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:57:50.513701 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_c42c4d9b6c1e4d2abd6aa51f251be773 [id=a72d39e753b2435a800e68d308cdc86c]
I20260924 19:57:50.431211 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 97a1b8ccd35a43cea7a71d705fad1494 (table default.Table_850e5158d78d4850b87445dac75ef7a9 [id=f768ce157a7b407d9f1e55a5b9d0720a]) successfully deleted
W20260924 19:57:50.519500 17599 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 88813.5us
W20260924 19:57:50.520143 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 9c861e93ad9f40ba9ad010021e43bc2b because the tablet was not found. No further retry: Not found: Tablet not found: 9c861e93ad9f40ba9ad010021e43bc2b
I20260924 19:57:50.521564 19369 ts_tablet_manager.cc:1929] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:57:50.521975 19369 log.cc:1199] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/57008266b80345e0a3f8e60bd7e2d62b
I20260924 19:57:50.522742 19369 ts_tablet_manager.cc:1950] T 57008266b80345e0a3f8e60bd7e2d62b P c4f6e3a7b06148918d230c77ab0c8202: Deleting consensus metadata
I20260924 19:57:50.542208 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06]
I20260924 19:57:50.546656 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]
I20260924 19:57:50.550868 18224 tablet_bootstrap.cc:492] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:50.551349 18224 ts_tablet_manager.cc:1403] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.416s user 0.009s sys 0.008s
I20260924 19:57:50.553653 18224 raft_consensus.cc:359] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.554242 18224 raft_consensus.cc:385] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:50.554504 18224 raft_consensus.cc:740] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:50.567303 19204 raft_consensus.cc:493] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:50.567770 19204 raft_consensus.cc:515] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:50.571266 19204 leader_election.cc:290] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:50.584112 18050 leader_election.cc:304] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:50.585705 19403 raft_consensus.cc:2804] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:50.587270 19403 raft_consensus.cc:697] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:50.586769 17905 leader_election.cc:304] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:50.592383 18051 leader_election.cc:304] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
W20260924 19:57:50.599984 19431 log.cc:927] Time spent T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35: Append to log took a long time: real 0.056s user 0.007s sys 0.018s
I20260924 19:57:50.600528 18223 tablet_bootstrap.cc:492] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:50.601004 18223 ts_tablet_manager.cc:1403] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.281s user 0.012s sys 0.002s
I20260924 19:57:50.588085 19403 consensus_queue.cc:237] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.611068 19402 raft_consensus.cc:2804] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:50.611603 19402 raft_consensus.cc:493] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:50.611953 19402 raft_consensus.cc:3060] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:50.648579 19402 raft_consensus.cc:515] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.687512 18224 consensus_queue.cc:260] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.692565 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfe0303596204e528585c4fe1c476d8b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:50.697085 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_4e897947eb9e4466a9f050091b6a7b60 [id=b7032f43d384487d9b041986780669c0]
I20260924 19:57:50.709990 19435 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6f88e00b235f48bb82e6a3c9c9f51fa6)
I20260924 19:57:50.714811 18224 ts_tablet_manager.cc:1434] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.163s user 0.004s sys 0.001s
I20260924 19:57:50.693233 17815 raft_consensus.cc:2468] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:50.721130 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_954dba173a70455b90daee9230b44b6f [id=b743c3e3916a45069fabeed784e9e249]
I20260924 19:57:50.736552 18223 raft_consensus.cc:359] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.737242 18223 raft_consensus.cc:385] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:50.737517 18223 raft_consensus.cc:740] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:50.660786 19336 raft_consensus.cc:2749] T e1137318a9fa4a0781f718dc34c4ecbe P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:50.742766 18223 consensus_queue.cc:260] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:50.650647 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 64426.5us
I20260924 19:57:50.635995 19433 raft_consensus.cc:493] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:50.749472 19402 leader_election.cc:290] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:50.751524 18223 ts_tablet_manager.cc:1434] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.150s user 0.008s sys 0.000s
I20260924 19:57:50.753266 18223 tablet_bootstrap.cc:492] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:50.761932 19336 consensus_queue.cc:1048] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:50.762642 18220 tablet_bootstrap.cc:492] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:50.764695 18220 ts_tablet_manager.cc:1403] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.517s user 0.043s sys 0.020s
I20260924 19:57:50.766172 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_14bf9267f2da4ba48354027fc92cab3d [id=b898f46c59b246e39fe8b62a45019ecb]
I20260924 19:57:50.710841 19435 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:50.779727 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d930e7f4a6b7471899dab1b28e40f93d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:50.767931 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_415dca00ed274e59961ddf45e42de178 [id=b8a8b30617164fbcb9f6918142768362]
I20260924 19:57:50.792325 19398 raft_consensus.cc:493] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:50.792861 19398 raft_consensus.cc:515] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.793906 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_b8cbd748636249c0ab17657b8f259893 [id=b9bee2849a334d098dd5d52fed7f6cc0]
I20260924 19:57:50.795846 19398 leader_election.cc:290] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:50.796180 17970 raft_consensus.cc:3060] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:47.080674 18939 rpcz_store.cc:276] 0924 19:57:41.549848 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:41.550028 (+ 180us) service_pool.cc:224] Handling call
0924 19:57:44.816109 (+3266081us) raft_consensus.cc:1471] Updating replica for 243 ops
0924 19:57:44.913345 (+ 97236us) raft_consensus.cc:1534] Early marking committed up to index 6
0924 19:57:44.934640 (+ 21295us) raft_consensus.cc:1539] Triggering prepare for 243 ops
0924 19:57:44.935119 (+ 479us) write_op.cc:270] Start()
0924 19:57:44.935272 (+ 153us) write_op.cc:276] Timestamp: P: 1790279843473334 usec, L: 0
0924 19:57:44.935987 (+ 715us) write_op.cc:270] Start()
0924 19:57:44.936110 (+ 123us) write_op.cc:276] Timestamp: P: 1790279843479105 usec, L: 0
0924 19:57:44.936706 (+ 596us) write_op.cc:270] Start()
0924 19:57:44.936789 (+ 83us) write_op.cc:276] Timestamp: P: 1790279843488638 usec, L: 0
0924 19:57:44.937291 (+ 502us) write_op.cc:270] Start()
0924 19:57:44.937372 (+ 81us) write_op.cc:276] Timestamp: P: 1790279843528506 usec, L: 0
0924 19:57:44.937999 (+ 627us) write_op.cc:270] Start()
0924 19:57:44.938100 (+ 101us) write_op.cc:276] Timestamp: P: 1790279843539617 usec, L: 0
0924 19:57:44.938620 (+ 520us) write_op.cc:270] Start()
0924 19:57:44.938699 (+ 79us) write_op.cc:276] Timestamp: P: 1790279843556113 usec, L: 0
0924 19:57:44.986341 (+ 47642us) write_op.cc:270] Start()
0924 19:57:44.986538 (+ 197us) write_op.cc:276] Timestamp: P: 1790279843569619 usec, L: 0
0924 19:57:44.987077 (+ 539us) write_op.cc:270] Start()
0924 19:57:44.987170 (+ 93us) write_op.cc:276] Timestamp: P: 1790279843595882 usec, L: 0
0924 19:57:44.987667 (+ 497us) write_op.cc:270] Start()
0924 19:57:44.987744 (+ 77us) write_op.cc:276] Timestamp: P: 1790279843686355 usec, L: 0
0924 19:57:44.988245 (+ 501us) write_op.cc:270] Start()
0924 19:57:44.988317 (+ 72us) write_op.cc:276] Timestamp: P: 1790279843843081 usec, L: 0
0924 19:57:44.988800 (+ 483us) write_op.cc:270] Start()
0924 19:57:44.988901 (+ 101us) write_op.cc:276] Timestamp: P: 1790279843863418 usec, L: 0
0924 19:57:44.989748 (+ 847us) write_op.cc:270] Start()
0924 19:57:44.989835 (+ 87us) write_op.cc:276] Timestamp: P: 1790279843887422 usec, L: 0
0924 19:57:44.990296 (+ 461us) write_op.cc:270] Start()
0924 19:57:44.990371 (+ 75us) write_op.cc:276] Timestamp: P: 1790279843944102 usec, L: 0
0924 19:57:45.019591 (+ 29220us) write_op.cc:270] Start()
0924 19:57:45.019698 (+ 107us) write_op.cc:276] Timestamp: P: 1790279843969181 usec, L: 0
0924 19:57:45.020253 (+ 555us) write_op.cc:270] Start()
0924 19:57:45.020341 (+ 88us) write_op.cc:276] Timestamp: P: 1790279843998108 usec, L: 0
0924 19:57:45.020850 (+ 509us) write_op.cc:270] Start()
0924 19:57:45.045438 (+ 24588us) write_op.cc:276] Timestamp: P: 1790279844035689 usec, L: 0
0924 19:57:45.046090 (+ 652us) write_op.cc:270] Start()
0924 19:57:45.046186 (+ 96us) write_op.cc:276] Timestamp: P: 1790279844064878 usec, L: 0
0924 19:57:45.046670 (+ 484us) write_op.cc:270] Start()
0924 19:57:45.046751 (+ 81us) write_op.cc:276] Timestamp: P: 1790279844184963 usec, L: 0
0924 19:57:45.052274 (+ 5523us) write_op.cc:270] Start()
0924 19:57:45.052452 (+ 178us) write_op.cc:276] Timestamp: P: 1790279844188642 usec, L: 0
0924 19:57:45.052960 (+ 508us) write_op.cc:270] Start()
0924 19:57:45.053042 (+ 82us) write_op.cc:276] Timestamp: P: 1790279844263377 usec, L: 0
0924 19:57:45.053503 (+ 461us) write_op.cc:270] Start()
0924 19:57:45.053576 (+ 73us) write_op.cc:276] Timestamp: P: 1790279844317755 usec, L: 0
0924 19:57:45.054023 (+ 447us) write_op.cc:270] Start()
0924 19:57:45.054101 (+ 78us) write_op.cc:276] Timestamp: P: 1790279844375893 usec, L: 0
0924 19:57:45.054563 (+ 462us) write_op.cc:270] Start()
0924 19:57:45.054680 (+ 117us) write_op.cc:276] Timestamp: P: 1790279844380405 usec, L: 0
0924 19:57:45.055127 (+ 447us) write_op.cc:270] Start()
0924 19:57:45.079315 (+ 24188us) write_op.cc:276] Timestamp: P: 1790279844430608 usec, L: 0
0924 19:57:45.080069 (+ 754us) write_op.cc:270] Start()
0924 19:57:45.080172 (+ 103us) write_op.cc:276] Timestamp: P: 1790279844475243 usec, L: 0
0924 19:57:45.082936 (+ 2764us) write_op.cc:270] Start()
0924 19:57:45.083033 (+ 97us) write_op.cc:276] Timestamp: P: 1790279844509346 usec, L: 0
0924 19:57:45.083604 (+ 571us) write_op.cc:270] Start()
0924 19:57:45.083694 (+ 90us) write_op.cc:276] Timestamp: P: 1790279844568838 usec, L: 0
0924 19:57:45.084207 (+ 513us) write_op.cc:270] Start()
0924 19:57:45.084291 (+ 84us) write_op.cc:276] Timestamp: P: 1790279844616743 usec, L: 0
0924 19:57:45.084755 (+ 464us) write_op.cc:270] Start()
0924 19:57:45.084830 (+ 75us) write_op.cc:276] Timestamp: P: 1790279844646456 usec, L: 0
0924 19:57:45.085342 (+ 512us) write_op.cc:270] Start()
0924 19:57:45.085444 (+ 102us) write_op.cc:276] Timestamp: P: 1790279844725342 usec, L: 0
0924 19:57:45.085992 (+ 548us) write_op.cc:270] Start()
0924 19:57:45.086079 (+ 87us) write_op.cc:276] Timestamp: P: 1790279844799307 usec, L: 0
0924 19:57:45.086598 (+ 519us) write_op.cc:270] Start()
0924 19:57:45.086680 (+ 82us) write_op.cc:276] Timestamp: P: 1790279844960800 usec, L: 0
0924 19:57:45.251588 (+164908us) write_op.cc:270] Start()
0924 19:57:45.251704 (+ 116us) write_op.cc:276] Timestamp: P: 1790279845046402 usec, L: 0
0924 19:57:45.252379 (+ 675us) write_op.cc:270] Start()
0924 19:57:45.252466 (+ 87us) write_op.cc:276] Timestamp: P: 1790279845051665 usec, L: 0
0924 19:57:45.253067 (+ 601us) write_op.cc:270] Start()
0924 19:57:45.253163 (+ 96us) write_op.cc:276] Timestamp: P: 1790279845129311 usec, L: 0
0924 19:57:45.253634 (+ 471us) write_op.cc:270] Start()
0924 19:57:45.253730 (+ 96us) write_op.cc:276] Timestamp: P: 1790279845140011 usec, L: 0
0924 19:57:45.254258 (+ 528us) write_op.cc:270] Start()
0924 19:57:45.254340 (+ 82us) write_op.cc:276] Timestamp: P: 1790279845142357 usec, L: 0
0924 19:57:45.254810 (+ 470us) write_op.cc:270] Start()
0924 19:57:45.254883 (+ 73us) write_op.cc:276] Timestamp: P: 1790279845156073 usec, L: 0
0924 19:57:45.267334 (+ 12451us) write_op.cc:270] Start()
0924 19:57:45.267445 (+ 111us) write_op.cc:276] Timestamp: P: 1790279845158200 usec, L: 0
0924 19:57:45.268030 (+ 585us) write_op.cc:270] Start()
0924 19:57:45.268131 (+ 101us) write_op.cc:276] Timestamp: P: 1790279845160538 usec, L: 0
0924 19:57:45.268616 (+ 485us) write_op.cc:270] Start()
0924 19:57:45.268699 (+ 83us) write_op.cc:276] Timestamp: P: 1790279845272977 usec, L: 0
0924 19:57:45.269198 (+ 499us) write_op.cc:270] Start()
0924 19:57:45.269284 (+ 86us) write_op.cc:276] Timestamp: P: 1790279845366758 usec, L: 0
0924 19:57:45.269790 (+ 506us) write_op.cc:270] Start()
0924 19:57:45.269874 (+ 84us) write_op.cc:276] Timestamp: P: 1790279845388870 usec, L: 0
0924 19:57:45.270363 (+ 489us) write_op.cc:270] Start()
0924 19:57:45.270444 (+ 81us) write_op.cc:276] Timestamp: P: 1790279845413192 usec, L: 0
0924 19:57:45.270896 (+ 452us) write_op.cc:270] Start()
0924 19:57:45.270984 (+ 88us) write_op.cc:276] Timestamp: P: 1790279845515888 usec, L: 0
0924 19:57:45.305114 (+ 34130us) write_op.cc:270] Start()
0924 19:57:45.305212 (+ 98us) write_op.cc:276] Timestamp: P: 1790279845576662 usec, L: 0
0924 19:57:45.305780 (+ 568us) write_op.cc:270] Start()
0924 19:57:45.305870 (+ 90us) write_op.cc:276] Timestamp: P: 1790279845653181 usec, L: 0
0924 19:57:45.306401 (+ 531us) write_op.cc:270] Start()
0924 19:57:45.306596 (+ 195us) write_op.cc:276] Timestamp: P: 1790279845719498 usec, L: 0
0924 19:57:45.307241 (+ 645us) write_op.cc:270] Start()
0924 19:57:45.307347 (+ 106us) write_op.cc:276] Timestamp: P: 1790279845780156 usec, L: 0
0924 19:57:45.338375 (+ 31028us) write_op.cc:270] Start()
0924 19:57:45.338481 (+ 106us) write_op.cc:276] Timestamp: P: 1790279845792953 usec, L: 0
0924 19:57:45.339067 (+ 586us) write_op.cc:270] Start()
0924 19:57:45.339178 (+ 111us) write_op.cc:276] Timestamp: P: 1790279845797295 usec, L: 0
0924 19:57:45.339669 (+ 491us) write_op.cc:270] Start()
0924 19:57:45.339742 (+ 73us) write_op.cc:276] Timestamp: P: 1790279845829321 usec, L: 0
0924 19:57:45.340213 (+ 471us) write_op.cc:270] Start()
0924 19:57:45.340288 (+ 75us) write_op.cc:276] Timestamp: P: 1790279845836297 usec, L: 0
0924 19:57:45.340742 (+ 454us) write_op.cc:270] Start()
0924 19:57:45.340816 (+ 74us) write_op.cc:276] Timestamp: P: 1790279845850636 usec, L: 0
0924 19:57:45.341314 (+ 498us) write_op.cc:270] Start()
0924 19:57:45.341404 (+ 90us) write_op.cc:276] Timestamp: P: 1790279845864889 usec, L: 0
0924 19:57:45.341878 (+ 474us) write_op.cc:270] Start()
0924 19:57:45.341969 (+ 91us) write_op.cc:276] Timestamp: P: 1790279845952623 usec, L: 0
0924 19:57:45.342435 (+ 466us) write_op.cc:270] Start()
0924 19:57:45.342514 (+ 79us) write_op.cc:276] Timestamp: P: 1790279845964096 usec, L: 0
0924 19:57:45.381176 (+ 38662us) write_op.cc:270] Start()
0924 19:57:45.381298 (+ 122us) write_op.cc:276] Timestamp: P: 1790279846004296 usec, L: 0
0924 19:57:45.381992 (+ 694us) write_op.cc:270] Start()
0924 19:57:45.382077 (+ 85us) write_op.cc:276] Timestamp: P: 1790279846030067 usec, L: 0
0924 19:57:45.382580 (+ 503us) write_op.cc:270] Start()
0924 19:57:45.382748 (+ 168us) write_op.cc:276] Timestamp: P: 1790279846059886 usec, L: 0
0924 19:57:45.383245 (+ 497us) write_op.cc:270] Start()
0924 19:57:45.383331 (+ 86us) write_op.cc:276] Timestamp: P: 1790279846075495 usec, L: 0
0924 19:57:45.383820 (+ 489us) write_op.cc:270] Start()
0924 19:57:45.383899 (+ 79us) write_op.cc:276] Timestamp: P: 1790279846078171 usec, L: 0
0924 19:57:45.384539 (+ 640us) write_op.cc:270] Start()
0924 19:57:45.384614 (+ 75us) write_op.cc:276] Timestamp: P: 1790279846113374 usec, L: 0
0924 19:57:45.385153 (+ 539us) write_op.cc:270] Start()
0924 19:57:45.385234 (+ 81us) write_op.cc:276] Timestamp: P: 1790279846136830 usec, L: 0
0924 19:57:45.385795 (+ 561us) write_op.cc:270] Start()
0924 19:57:45.385865 (+ 70us) write_op.cc:276] Timestamp: P: 1790279846228132 usec, L: 0
0924 19:57:45.435727 (+ 49862us) write_op.cc:270] Start()
0924 19:57:45.435856 (+ 129us) write_op.cc:276] Timestamp: P: 1790279846381708 usec, L: 0
0924 19:57:45.436436 (+ 580us) write_op.cc:270] Start()
0924 19:57:45.436522 (+ 86us) write_op.cc:276] Timestamp: P: 1790279846389406 usec, L: 0
0924 19:57:45.437031 (+ 509us) write_op.cc:270] Start()
0924 19:57:45.437114 (+ 83us) write_op.cc:276] Timestamp: P: 1790279846393244 usec, L: 0
0924 19:57:45.437616 (+ 502us) write_op.cc:270] Start()
0924 19:57:45.437699 (+ 83us) write_op.cc:276] Timestamp: P: 1790279846486276 usec, L: 0
0924 19:57:45.438318 (+ 619us) write_op.cc:270] Start()
0924 19:57:45.438402 (+ 84us) write_op.cc:276] Timestamp: P: 1790279846499899 usec, L: 0
0924 19:57:45.438856 (+ 454us) write_op.cc:270] Start()
0924 19:57:45.438938 (+ 82us) write_op.cc:276] Timestamp: P: 1790279846501957 usec, L: 0
0924 19:57:45.439426 (+ 488us) write_op.cc:270] Start()
0924 19:57:45.439499 (+ 73us) write_op.cc:276] Timestamp: P: 1790279846527059 usec, L: 0
0924 19:57:45.439961 (+ 462us) write_op.cc:270] Start()
0924 19:57:45.440047 (+ 86us) write_op.cc:276] Timestamp: P: 1790279846594412 usec, L: 0
0924 19:57:45.440520 (+ 473us) write_op.cc:270] Start()
0924 19:57:45.440601 (+ 81us) write_op.cc:276] Timestamp: P: 1790279846771572 usec, L: 0
0924 19:57:45.441097 (+ 496us) write_op.cc:270] Start()
0924 19:57:45.441172 (+ 75us) write_op.cc:276] Timestamp: P: 1790279846789829 usec, L: 0
0924 19:57:45.441614 (+ 442us) write_op.cc:270] Start()
0924 19:57:45.441692 (+ 78us) write_op.cc:276] Timestamp: P: 1790279846895754 usec, L: 0
0924 19:57:45.442172 (+ 480us) write_op.cc:270] Start()
0924 19:57:45.442246 (+ 74us) write_op.cc:276] Timestamp: P: 1790279846922182 usec, L: 0
0924 19:57:45.442705 (+ 459us) write_op.cc:270] Start()
0924 19:57:45.442780 (+ 75us) write_op.cc:276] Timestamp: P: 1790279846926522 usec, L: 0
0924 19:57:45.477806 (+ 35026us) write_op.cc:270] Start()
0924 19:57:45.477915 (+ 109us) write_op.cc:276] Timestamp: P: 1790279847035608 usec, L: 0
0924 19:57:45.478509 (+ 594us) write_op.cc:270] Start()
0924 19:57:45.579797 (+101288us) write_op.cc:276] Timestamp: P: 1790279847114643 usec, L: 0
0924 19:57:45.612194 (+ 32397us) write_op.cc:270] Start()
0924 19:57:45.612390 (+ 196us) write_op.cc:276] Timestamp: P: 1790279847134265 usec, L: 0
0924 19:57:45.613603 (+ 1213us) write_op.cc:270] Start()
0924 19:57:45.613842 (+ 239us) write_op.cc:276] Timestamp: P: 1790279847151309 usec, L: 0
0924 19:57:45.615799 (+ 1957us) write_op.cc:270] Start()
0924 19:57:45.616126 (+ 327us) write_op.cc:276] Timestamp: P: 1790279847219907 usec, L: 0
0924 19:57:45.618324 (+ 2198us) write_op.cc:270] Start()
0924 19:57:45.618684 (+ 360us) write_op.cc:276] Timestamp: P: 1790279847276824 usec, L: 0
0924 19:57:45.620904 (+ 2220us) write_op.cc:270] Start()
0924 19:57:45.621353 (+ 449us) write_op.cc:276] Timestamp: P: 1790279847329278 usec, L: 0
0924 19:57:45.624649 (+ 3296us) write_op.cc:270] Start()
0924 19:57:45.625122 (+ 473us) write_op.cc:276] Timestamp: P: 1790279847333412 usec, L: 0
0924 19:57:45.739274 (+114152us) write_op.cc:270] Start()
0924 19:57:45.739538 (+ 264us) write_op.cc:276] Timestamp: P: 1790279847412738 usec, L: 0
0924 19:57:45.740142 (+ 604us) write_op.cc:270] Start()
0924 19:57:45.740227 (+ 85us) write_op.cc:276] Timestamp: P: 1790279847493563 usec, L: 0
0924 19:57:45.741031 (+ 804us) write_op.cc:270] Start()
0924 19:57:45.741113 (+ 82us) write_op.cc:276] Timestamp: P: 1790279847556074 usec, L: 0
0924 19:57:45.741575 (+ 462us) write_op.cc:270] Start()
0924 19:57:45.741652 (+ 77us) write_op.cc:276] Timestamp: P: 1790279847664640 usec, L: 0
0924 19:57:45.742404 (+ 752us) write_op.cc:270] Start()
0924 19:57:45.742486 (+ 82us) write_op.cc:276] Timestamp: P: 1790279847666953 usec, L: 0
0924 19:57:45.742934 (+ 448us) write_op.cc:270] Start()
0924 19:57:45.743005 (+ 71us) write_op.cc:276] Timestamp: P: 1790279847720352 usec, L: 0
0924 19:57:45.743452 (+ 447us) write_op.cc:270] Start()
0924 19:57:45.793152 (+ 49700us) write_op.cc:276] Timestamp: P: 1790279847803508 usec, L: 0
0924 19:57:45.793786 (+ 634us) write_op.cc:270] Start()
0924 19:57:45.793871 (+ 85us) write_op.cc:276] Timestamp: P: 1790279847806412 usec, L: 0
0924 19:57:45.794324 (+ 453us) write_op.cc:270] Start()
0924 19:57:45.794418 (+ 94us) write_op.cc:276] Timestamp: P: 1790279847809484 usec, L: 0
0924 19:57:45.795002 (+ 584us) write_op.cc:270] Start()
0924 19:57:45.795075 (+ 73us) write_op.cc:276] Timestamp: P: 1790279847904117 usec, L: 0
0924 19:57:45.795605 (+ 530us) write_op.cc:270] Start()
0924 19:57:45.795673 (+ 68us) write_op.cc:276] Timestamp: P: 1790279847944055 usec, L: 0
0924 19:57:45.796115 (+ 442us) write_op.cc:270] Start()
0924 19:57:45.796294 (+ 179us) write_op.cc:276] Timestamp: P: 1790279847946140 usec, L: 0
0924 19:57:45.796842 (+ 548us) write_op.cc:270] Start()
0924 19:57:45.796911 (+ 69us) write_op.cc:276] Timestamp: P: 1790279847989398 usec, L: 0
0924 19:57:45.797353 (+ 442us) write_op.cc:270] Start()
0924 19:57:45.797421 (+ 68us) write_op.cc:276] Timestamp: P: 1790279848008162 usec, L: 0
0924 19:57:45.797850 (+ 429us) write_op.cc:270] Start()
0924 19:57:45.797920 (+ 70us) write_op.cc:276] Timestamp: P: 1790279848047444 usec, L: 0
0924 19:57:45.888530 (+ 90610us) write_op.cc:270] Start()
0924 19:57:45.907187 (+ 18657us) write_op.cc:276] Timestamp: P: 1790279848066420 usec, L: 0
0924 19:57:45.913937 (+ 6750us) write_op.cc:270] Start()
0924 19:57:45.914033 (+ 96us) write_op.cc:276] Timestamp: P: 1790279848152823 usec, L: 0
0924 19:57:45.914600 (+ 567us) write_op.cc:270] Start()
0924 19:57:45.914680 (+ 80us) write_op.cc:276] Timestamp: P: 1790279848234958 usec, L: 0
0924 19:57:45.915129 (+ 449us) write_op.cc:270] Start()
0924 19:57:45.915232 (+ 103us) write_op.cc:276] Timestamp: P: 1790279848299748 usec, L: 0
0924 19:57:45.915667 (+ 435us) write_op.cc:270] Start()
0924 19:57:45.915749 (+ 82us) write_op.cc:276] Timestamp: P: 1790279848313052 usec, L: 0
0924 19:57:45.916177 (+ 428us) write_op.cc:270] Start()
0924 19:57:45.916249 (+ 72us) write_op.cc:276] Timestamp: P: 1790279848324319 usec, L: 0
0924 19:57:45.916713 (+ 464us) write_op.cc:270] Start()
0924 19:57:45.916781 (+ 68us) write_op.cc:276] Timestamp: P: 1790279848356980 usec, L: 0
0924 19:57:45.950520 (+ 33739us) write_op.cc:270] Start()
0924 19:57:45.950614 (+ 94us) write_op.cc:276] Timestamp: P: 1790279848387813 usec, L: 0
0924 19:57:45.980073 (+ 29459us) write_op.cc:270] Start()
0924 19:57:45.980170 (+ 97us) write_op.cc:276] Timestamp: P: 1790279848560491 usec, L: 0
0924 19:57:45.980957 (+ 787us) write_op.cc:270] Start()
0924 19:57:45.981034 (+ 77us) write_op.cc:276] Timestamp: P: 1790279848562470 usec, L: 0
0924 19:57:45.981613 (+ 579us) write_op.cc:270] Start()
0924 19:57:45.981695 (+ 82us) write_op.cc:276] Timestamp: P: 1790279848600913 usec, L: 0
0924 19:57:45.982097 (+ 402us) write_op.cc:270] Start()
0924 19:57:45.982162 (+ 65us) write_op.cc:276] Timestamp: P: 1790279848609817 usec, L: 0
0924 19:57:45.982564 (+ 402us) write_op.cc:270] Start()
0924 19:57:45.982629 (+ 65us) write_op.cc:276] Timestamp: P: 1790279848711310 usec, L: 0
0924 19:57:45.983023 (+ 394us) write_op.cc:270] Start()
0924 19:57:45.983088 (+ 65us) write_op.cc:276] Timestamp: P: 1790279848714477 usec, L: 0
0924 19:57:45.987369 (+ 4281us) write_op.cc:270] Start()
0924 19:57:45.987447 (+ 78us) write_op.cc:276] Timestamp: P: 1790279848723261 usec, L: 0
0924 19:57:45.988119 (+ 672us) write_op.cc:270] Start()
0924 19:57:45.988185 (+ 66us) write_op.cc:276] Timestamp: P: 1790279848911044 usec, L: 0
0924 19:57:45.988693 (+ 508us) write_op.cc:270] Start()
0924 19:57:45.988772 (+ 79us) write_op.cc:276] Timestamp: P: 1790279848927642 usec, L: 0
0924 19:57:45.989188 (+ 416us) write_op.cc:270] Start()
0924 19:57:45.989256 (+ 68us) write_op.cc:276] Timestamp: P: 1790279848945561 usec, L: 0
0924 19:57:45.989707 (+ 451us) write_op.cc:270] Start()
0924 19:57:45.989775 (+ 68us) write_op.cc:276] Timestamp: P: 1790279848981816 usec, L: 0
0924 19:57:45.990199 (+ 424us) write_op.cc:270] Start()
0924 19:57:45.990273 (+ 74us) write_op.cc:276] Timestamp: P: 1790279848983781 usec, L: 0
0924 19:57:45.990692 (+ 419us) write_op.cc:270] Start()
0924 19:57:45.990764 (+ 72us) write_op.cc:276] Timestamp: P: 1790279849056443 usec, L: 0
0924 19:57:46.032575 (+ 41811us) write_op.cc:270] Start()
0924 19:57:46.032675 (+ 100us) write_op.cc:276] Timestamp: P: 1790279849109964 usec, L: 0
0924 19:57:46.034156 (+ 1481us) write_op.cc:270] Start()
0924 19:57:46.034946 (+ 790us) write_op.cc:276] Timestamp: P: 1790279849123861 usec, L: 0
0924 19:57:46.035426 (+ 480us) write_op.cc:270] Start()
0924 19:57:46.035515 (+ 89us) write_op.cc:276] Timestamp: P: 1790279849269629 usec, L: 0
0924 19:57:46.065287 (+ 29772us) write_op.cc:270] Start()
0924 19:57:46.065368 (+ 81us) write_op.cc:276] Timestamp: P: 1790279849273203 usec, L: 0
0924 19:57:46.065846 (+ 478us) write_op.cc:270] Start()
0924 19:57:46.065917 (+ 71us) write_op.cc:276] Timestamp: P: 1790279849276635 usec, L: 0
0924 19:57:46.119514 (+ 53597us) write_op.cc:270] Start()
0924 19:57:46.119607 (+ 93us) write_op.cc:276] Timestamp: P: 1790279849278648 usec, L: 0
0924 19:57:46.120200 (+ 593us) write_op.cc:270] Start()
0924 19:57:46.120268 (+ 68us) write_op.cc:276] Timestamp: P: 1790279849391594 usec, L: 0
0924 19:57:46.120696 (+ 428us) write_op.cc:270] Start()
0924 19:57:46.120758 (+ 62us) write_op.cc:276] Timestamp: P: 1790279849443193 usec, L: 0
0924 19:57:46.121185 (+ 427us) write_op.cc:270] Start()
0924 19:57:46.121337 (+ 152us) write_op.cc:276] Timestamp: P: 1790279849466191 usec, L: 0
0924 19:57:46.121778 (+ 441us) write_op.cc:270] Start()
0924 19:57:46.121841 (+ 63us) write_op.cc:276] Timestamp: P: 1790279849490729 usec, L: 0
0924 19:57:46.122248 (+ 407us) write_op.cc:270] Start()
0924 19:57:46.122314 (+ 66us) write_op.cc:276] Timestamp: P: 1790279849566932 usec, L: 0
0924 19:57:46.122721 (+ 407us) write_op.cc:270] Start()
0924 19:57:46.122788 (+ 67us) write_op.cc:276] Timestamp: P: 1790279849636239 usec, L: 0
0924 19:57:46.123143 (+ 355us) write_op.cc:270] Start()
0924 19:57:46.183315 (+ 60172us) write_op.cc:276] Timestamp: P: 1790279849638204 usec, L: 0
0924 19:57:46.183884 (+ 569us) write_op.cc:270] Start()
0924 19:57:46.183954 (+ 70us) write_op.cc:276] Timestamp: P: 1790279849639972 usec, L: 0
0924 19:57:46.184382 (+ 428us) write_op.cc:270] Start()
0924 19:57:46.184459 (+ 77us) write_op.cc:276] Timestamp: P: 1790279849658340 usec, L: 0
0924 19:57:46.184871 (+ 412us) write_op.cc:270] Start()
0924 19:57:46.184941 (+ 70us) write_op.cc:276] Timestamp: P: 1790279849679549 usec, L: 0
0924 19:57:46.185362 (+ 421us) write_op.cc:270] Start()
0924 19:57:46.185443 (+ 81us) write_op.cc:276] Timestamp: P: 1790279849723037 usec, L: 0
0924 19:57:46.185856 (+ 413us) write_op.cc:270] Start()
0924 19:57:46.185925 (+ 69us) write_op.cc:276] Timestamp: P: 1790279849883392 usec, L: 0
0924 19:57:46.186326 (+ 401us) write_op.cc:270] Start()
0924 19:57:46.186391 (+ 65us) write_op.cc:276] Timestamp: P: 1790279849952589 usec, L: 0
0924 19:57:46.186840 (+ 449us) write_op.cc:270] Start()
0924 19:57:46.186911 (+ 71us) write_op.cc:276] Timestamp: P: 1790279850122482 usec, L: 0
0924 19:57:46.215420 (+ 28509us) write_op.cc:270] Start()
0924 19:57:46.215519 (+ 99us) write_op.cc:276] Timestamp: P: 1790279850146379 usec, L: 0
0924 19:57:46.216577 (+ 1058us) write_op.cc:270] Start()
0924 19:57:46.216667 (+ 90us) write_op.cc:276] Timestamp: P: 1790279850152347 usec, L: 0
0924 19:57:46.217127 (+ 460us) write_op.cc:270] Start()
0924 19:57:46.217195 (+ 68us) write_op.cc:276] Timestamp: P: 1790279850280848 usec, L: 0
0924 19:57:46.217634 (+ 439us) write_op.cc:270] Start()
0924 19:57:46.217710 (+ 76us) write_op.cc:276] Timestamp: P: 1790279850394200 usec, L: 0
0924 19:57:46.218191 (+ 481us) write_op.cc:270] Start()
0924 19:57:46.218259 (+ 68us) write_op.cc:276] Timestamp: P: 1790279850397009 usec, L: 0
0924 19:57:46.222134 (+ 3875us) write_op.cc:270] Start()
0924 19:57:46.222206 (+ 72us) write_op.cc:276] Timestamp: P: 1790279850399119 usec, L: 0
0924 19:57:46.222646 (+ 440us) write_op.cc:270] Start()
0924 19:57:46.222714 (+ 68us) write_op.cc:276] Timestamp: P: 1790279850469858 usec, L: 0
0924 19:57:46.223135 (+ 421us) write_op.cc:270] Start()
0924 19:57:46.223222 (+ 87us) write_op.cc:276] Timestamp: P: 1790279850593614 usec, L: 0
0924 19:57:46.223637 (+ 415us) write_op.cc:270] Start()
0924 19:57:46.223707 (+ 70us) write_op.cc:276] Timestamp: P: 1790279850730002 usec, L: 0
0924 19:57:46.256493 (+ 32786us) write_op.cc:270] Start()
0924 19:57:46.256587 (+ 94us) write_op.cc:276] Timestamp: P: 1790279850766794 usec, L: 0
0924 19:57:46.258015 (+ 1428us) write_op.cc:270] Start()
0924 19:57:46.258089 (+ 74us) write_op.cc:276] Timestamp: P: 1790279850880532 usec, L: 0
0924 19:57:46.258544 (+ 455us) write_op.cc:270] Start()
0924 19:57:46.258612 (+ 68us) write_op.cc:276] Timestamp: P: 1790279851013172 usec, L: 0
0924 19:57:46.259028 (+ 416us) write_op.cc:270] Start()
0924 19:57:46.259091 (+ 63us) write_op.cc:276] Timestamp: P: 1790279851110025 usec, L: 0
0924 19:57:46.259546 (+ 455us) write_op.cc:270] Start()
0924 19:57:46.259613 (+ 67us) write_op.cc:276] Timestamp: P: 1790279851265949 usec, L: 0
0924 19:57:46.260025 (+ 412us) write_op.cc:270] Start()
0924 19:57:46.260092 (+ 67us) write_op.cc:276] Timestamp: P: 1790279851553230 usec, L: 0
0924 19:57:46.260530 (+ 438us) write_op.cc:270] Start()
0924 19:57:46.260602 (+ 72us) write_op.cc:276] Timestamp: P: 1790279851679375 usec, L: 0
0924 19:57:46.261037 (+ 435us) write_op.cc:270] Start()
0924 19:57:46.261103 (+ 66us) write_op.cc:276] Timestamp: P: 1790279851742234 usec, L: 0
0924 19:57:46.261533 (+ 430us) write_op.cc:270] Start()
0924 19:57:46.261604 (+ 71us) write_op.cc:276] Timestamp: P: 1790279851844684 usec, L: 0
0924 19:57:46.290131 (+ 28527us) write_op.cc:270] Start()
0924 19:57:46.290221 (+ 90us) write_op.cc:276] Timestamp: P: 1790279852006377 usec, L: 0
0924 19:57:46.290720 (+ 499us) write_op.cc:270] Start()
0924 19:57:46.290790 (+ 70us) write_op.cc:276] Timestamp: P: 1790279852014562 usec, L: 0
0924 19:57:46.291220 (+ 430us) write_op.cc:270] Start()
0924 19:57:46.303999 (+ 12779us) write_op.cc:276] Timestamp: P: 1790279852086461 usec, L: 0
0924 19:57:46.304574 (+ 575us) write_op.cc:270] Start()
0924 19:57:46.304661 (+ 87us) write_op.cc:276] Timestamp: P: 1790279852256445 usec, L: 0
0924 19:57:46.305096 (+ 435us) write_op.cc:270] Start()
0924 19:57:46.305166 (+ 70us) write_op.cc:276] Timestamp: P: 1790279852282992 usec, L: 0
0924 19:57:46.305603 (+ 437us) write_op.cc:270] Start()
0924 19:57:46.305675 (+ 72us) write_op.cc:276] Timestamp: P: 1790279852370998 usec, L: 0
0924 19:57:46.306137 (+ 462us) write_op.cc:270] Start()
0924 19:57:46.306206 (+ 69us) write_op.cc:276] Timestamp: P: 1790279852445377 usec, L: 0
0924 19:57:46.306649 (+ 443us) write_op.cc:270] Start()
0924 19:57:46.306715 (+ 66us) write_op.cc:276] Timestamp: P: 1790279852479588 usec, L: 0
0924 19:57:46.315437 (+ 8722us) write_op.cc:270] Start()
0924 19:57:46.315518 (+ 81us) write_op.cc:276] Timestamp: P: 1790279852607002 usec, L: 0
0924 19:57:46.316023 (+ 505us) write_op.cc:270] Start()
0924 19:57:46.316098 (+ 75us) write_op.cc:276] Timestamp: P: 1790279852611530 usec, L: 0
0924 19:57:46.316562 (+ 464us) write_op.cc:270] Start()
0924 19:57:46.316640 (+ 78us) write_op.cc:276] Timestamp: P: 1790279852715193 usec, L: 0
0924 19:57:46.317083 (+ 443us) write_op.cc:270] Start()
0924 19:57:46.317153 (+ 70us) write_op.cc:276] Timestamp: P: 1790279852740561 usec, L: 0
0924 19:57:46.317575 (+ 422us) write_op.cc:270] Start()
0924 19:57:46.317648 (+ 73us) write_op.cc:276] Timestamp: P: 1790279852742648 usec, L: 0
0924 19:57:46.339942 (+ 22294us) write_op.cc:270] Start()
0924 19:57:46.340039 (+ 97us) write_op.cc:276] Timestamp: P: 1790279852779868 usec, L: 0
0924 19:57:46.340548 (+ 509us) write_op.cc:270] Start()
0924 19:57:46.340618 (+ 70us) write_op.cc:276] Timestamp: P: 1790279852793143 usec, L: 0
0924 19:57:46.341061 (+ 443us) write_op.cc:270] Start()
0924 19:57:46.341148 (+ 87us) write_op.cc:276] Timestamp: P: 1790279852825163 usec, L: 0
0924 19:57:46.341526 (+ 378us) write_op.cc:270] Start()
0924 19:57:46.341586 (+ 60us) write_op.cc:276] Timestamp: P: 1790279852854877 usec, L: 0
0924 19:57:46.341947 (+ 361us) write_op.cc:270] Start()
0924 19:57:46.342008 (+ 61us) write_op.cc:276] Timestamp: P: 1790279853111483 usec, L: 0
0924 19:57:46.342366 (+ 358us) write_op.cc:270] Start()
0924 19:57:46.342425 (+ 59us) write_op.cc:276] Timestamp: P: 1790279853177799 usec, L: 0
0924 19:57:46.342796 (+ 371us) write_op.cc:270] Start()
0924 19:57:46.342863 (+ 67us) write_op.cc:276] Timestamp: P: 1790279853186748 usec, L: 0
0924 19:57:46.343245 (+ 382us) write_op.cc:270] Start()
0924 19:57:46.357238 (+ 13993us) write_op.cc:276] Timestamp: P: 1790279853188989 usec, L: 0
0924 19:57:46.357765 (+ 527us) write_op.cc:270] Start()
0924 19:57:46.357845 (+ 80us) write_op.cc:276] Timestamp: P: 1790279853264111 usec, L: 0
0924 19:57:46.358271 (+ 426us) write_op.cc:270] Start()
0924 19:57:46.358336 (+ 65us) write_op.cc:276] Timestamp: P: 1790279853374364 usec, L: 0
0924 19:57:46.358762 (+ 426us) write_op.cc:270] Start()
0924 19:57:46.358829 (+ 67us) write_op.cc:276] Timestamp: P: 1790279853383193 usec, L: 0
0924 19:57:46.359315 (+ 486us) write_op.cc:270] Start()
0924 19:57:46.359381 (+ 66us) write_op.cc:276] Timestamp: P: 1790279853422417 usec, L: 0
0924 19:57:46.359858 (+ 477us) write_op.cc:270] Start()
0924 19:57:46.359927 (+ 69us) write_op.cc:276] Timestamp: P: 1790279853605571 usec, L: 0
0924 19:57:46.360365 (+ 438us) write_op.cc:270] Start()
0924 19:57:46.360448 (+ 83us) write_op.cc:276] Timestamp: P: 1790279853665566 usec, L: 0
0924 19:57:46.360880 (+ 432us) write_op.cc:270] Start()
0924 19:57:46.360963 (+ 83us) write_op.cc:276] Timestamp: P: 1790279853728330 usec, L: 0
0924 19:57:46.361386 (+ 423us) write_op.cc:270] Start()
0924 19:57:46.361464 (+ 78us) write_op.cc:276] Timestamp: P: 1790279853743377 usec, L: 0
0924 19:57:46.361869 (+ 405us) write_op.cc:270] Start()
0924 19:57:46.361931 (+ 62us) write_op.cc:276] Timestamp: P: 1790279853753991 usec, L: 0
0924 19:57:46.362326 (+ 395us) write_op.cc:270] Start()
0924 19:57:46.362392 (+ 66us) write_op.cc:276] Timestamp: P: 1790279853939828 usec, L: 0
0924 19:57:46.362878 (+ 486us) write_op.cc:270] Start()
0924 19:57:46.362943 (+ 65us) write_op.cc:276] Timestamp: P: 1790279854071580 usec, L: 0
0924 19:57:46.405994 (+ 43051us) write_op.cc:270] Start()
0924 19:57:46.406080 (+ 86us) write_op.cc:276] Timestamp: P: 1790279854157406 usec, L: 0
0924 19:57:46.406584 (+ 504us) write_op.cc:270] Start()
0924 19:57:46.406658 (+ 74us) write_op.cc:276] Timestamp: P: 1790279854167771 usec, L: 0
0924 19:57:46.407076 (+ 418us) write_op.cc:270] Start()
0924 19:57:46.415197 (+ 8121us) write_op.cc:276] Timestamp: P: 1790279854367030 usec, L: 0
0924 19:57:46.415778 (+ 581us) write_op.cc:270] Start()
0924 19:57:46.415856 (+ 78us) write_op.cc:276] Timestamp: P: 1790279854371318 usec, L: 0
0924 19:57:46.416263 (+ 407us) write_op.cc:270] Start()
0924 19:57:46.416327 (+ 64us) write_op.cc:276] Timestamp: P: 1790279
I20260924 19:57:50.797755 19433 raft_consensus.cc:515] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.803856 17967 raft_consensus.cc:3060] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:50.808148 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_a7d63063e7ee4493abe8c76da6d4f99d [id=be19c62729a34ae1a15966662303a869]
I20260924 19:57:50.809792 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_11ca95d108694c9498d4de67c9ff8808 [id=bec51c14454046329b3eb7016389f235]
I20260924 19:57:50.821337 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_a50ccb6bd6a042bc99b503e5aa1188ad [id=bfd5e6e55fb049f9965b190374b38c3d]
I20260924 19:57:50.823047 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_843b88f433f044458daa98228935a46e [id=c19508f168ed4191bdd25de1107ab416]
I20260924 19:57:50.823623 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfe0303596204e528585c4fe1c476d8b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:50.831266 19433 leader_election.cc:290] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:50.768142 19436 raft_consensus.cc:493] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:50.845242 17906 leader_election.cc:343] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 44199d2655f14dcf97afe2180789aae4
I20260924 19:57:50.825726 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_1d9dfc2ce03046ba955d3d82049402fd [id=c44dfe41598f4e6cbf320e3044b49fe5]
I20260924 19:57:50.848433 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:57:50.849215 19435 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:50.861476 17812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89646893588340038f30ed4754623ec2" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:50.842471 19436 raft_consensus.cc:515] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.864857 19436 leader_election.cc:290] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:50.869023 19369 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36562 (request call id 254) took 1364 ms. Trace:
I20260924 19:57:50.869447 19369 rpcz_store.cc:276] 0924 19:57:49.504412 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.504545 (+ 133us) service_pool.cc:224] Handling call
0924 19:57:50.129188 (+624643us) ts_tablet_manager.cc:1151] Deleting tablet 57008266b80345e0a3f8e60bd7e2d62b
0924 19:57:50.129221 (+ 33us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:50.256104 (+126883us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:50.521435 (+265331us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:50.868933 (+347498us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":624078}
I20260924 19:57:50.871913 18224 tablet_bootstrap.cc:492] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:50.875129 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c05ec2ba01448d998c6ffa1525de984" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:50.880122 17753 leader_election.cc:343] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:50.880124 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d930e7f4a6b7471899dab1b28e40f93d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:50.871035 17600 catalog_manager.cc:5063] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): tablet 57008266b80345e0a3f8e60bd7e2d62b (table default.Table_cca06980f109487a9ed8b8172950b60d [id=016f4039733144c985808b862eebcc9a]) successfully deleted
W20260924 19:57:50.884138 18050 leader_election.cc:343] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:50.885252 17600 catalog_manager.cc:5030] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): delete failed for tablet 97a1b8ccd35a43cea7a71d705fad1494 because the tablet was not found. No further retry: Not found: Tablet not found: 97a1b8ccd35a43cea7a71d705fad1494
I20260924 19:57:50.888046 18220 raft_consensus.cc:359] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.888748 18220 raft_consensus.cc:385] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:50.889015 18220 raft_consensus.cc:740] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:50.889652 18220 consensus_queue.cc:260] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.891634 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c05ec2ba01448d998c6ffa1525de984" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:50.775048 18223 tablet_bootstrap.cc:654] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:50.892943 17753 leader_election.cc:343] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:50.893316 17753 leader_election.cc:304] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:50.894229 18224 tablet_bootstrap.cc:654] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:50.894328 19436 raft_consensus.cc:2749] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:50.909267 18220 ts_tablet_manager.cc:1434] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.144s user 0.008s sys 0.000s
I20260924 19:57:50.911142 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_954c770a550c49599e38ee3850e4aea1 [id=c4ad22b552204e2d8209b275e57eb9c4]
I20260924 19:57:50.912693 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_bdae18e3011a421d96cdcc107e8f1dc2 [id=ca7affed79d94a2597fdaa1ff38e5e32]
I20260924 19:57:50.921378 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89646893588340038f30ed4754623ec2" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:50.919827 18220 tablet_bootstrap.cc:492] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:50.936926 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_fbc5ceeb4d1242d19344825cefb6ef2a [id=caf8f776b4754b80a6aac8211f1b7a6f]
I20260924 19:57:50.938464 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_8be3db507b2d42da90969bacc361c127 [id=cd5ff543cb1f48f58d5867aef9706963]
I20260924 19:57:50.939963 19398 raft_consensus.cc:493] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:50.940507 19398 raft_consensus.cc:515] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:50.942639 19398 leader_election.cc:290] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:50.953603 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df]
I20260924 19:57:50.954662 17815 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d930e7f4a6b7471899dab1b28e40f93d" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:50.955394 17815 raft_consensus.cc:3060] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:50.962272 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_bb9775fce761443ca3a306e0cace741e [id=dc2b2e2afb22446ba04263f4f86a23e0]
I20260924 19:57:50.964461 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_dea06619a5364df0961e47cb617c9e52 [id=e40a55e03a2d4ceea51d555d0c60a334]
W20260924 19:57:50.967469 17905 leader_election.cc:343] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 44199d2655f14dcf97afe2180789aae4
I20260924 19:57:50.967957 17905 leader_election.cc:304] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:50.968757 19336 raft_consensus.cc:2749] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:50.968966 17905 leader_election.cc:304] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
W20260924 19:57:50.970395 17905 consensus_peers.cc:597] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:50.970501 19336 raft_consensus.cc:2749] T 89646893588340038f30ed4754623ec2 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:50.981388 17815 raft_consensus.cc:2468] T d930e7f4a6b7471899dab1b28e40f93d P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
W20260924 19:57:50.983013 17906 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 8aef0359a77240679240823612507ee8
I20260924 19:57:50.986063 18220 tablet_bootstrap.cc:654] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:50.996470 17905 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 8aef0359a77240679240823612507ee8
I20260924 19:57:50.996917 17905 leader_election.cc:304] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:50.998755 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_83714385573a44b2af65ac658fbe2cfa [id=e5b211f0af7143709eec620ecf04f2b5]
I20260924 19:57:51.002272 19204 consensus_queue.cc:1048] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:51.003881 18051 consensus_peers.cc:597] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:51.005223 18051 leader_election.cc:304] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:51.006139 19204 raft_consensus.cc:2804] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:51.006451 19204 raft_consensus.cc:493] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:51.006793 19204 raft_consensus.cc:3060] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:51.010243 19398 raft_consensus.cc:2749] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:51.016654 18220 tablet_bootstrap.cc:492] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:51.017071 18220 ts_tablet_manager.cc:1403] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.010s
I20260924 19:57:51.024395 18220 raft_consensus.cc:359] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.025283 18220 raft_consensus.cc:385] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:51.025617 18220 raft_consensus.cc:740] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:51.028379 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_5128840101ca460b9e13dfd054764e9d [id=e8a1bc657d9b4fb2b1f0eb458d41dbbd]
W20260924 19:57:51.039983 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 67058.5us
I20260924 19:57:51.026432 18220 consensus_queue.cc:260] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.042665 19408 consensus_queue.cc:1048] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:51.043999 18220 ts_tablet_manager.cc:1434] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260924 19:57:51.044967 18220 tablet_bootstrap.cc:492] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
W20260924 19:57:51.045871 18050 leader_election.cc:343] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.047371 19408 consensus_queue.cc:1048] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:51.052599 18223 tablet_bootstrap.cc:492] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:51.056303 18220 tablet_bootstrap.cc:654] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:51.073190 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_0673b53c69bf4c429697ba739ef113e4 [id=ea252b103a0f43d18971c78fe2174c2e]
I20260924 19:57:51.085976 18224 tablet_bootstrap.cc:492] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:51.086505 18224 ts_tablet_manager.cc:1403] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.215s user 0.007s sys 0.004s
I20260924 19:57:51.088956 18224 raft_consensus.cc:359] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.089655 18224 raft_consensus.cc:385] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:51.089952 18224 raft_consensus.cc:740] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:51.091804 19450 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6f88e00b235f48bb82e6a3c9c9f51fa6)
I20260924 19:57:51.091655 18224 consensus_queue.cc:260] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.092805 19450 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:51.094273 18224 ts_tablet_manager.cc:1434] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260924 19:57:51.075037 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_3bf03540f35b468dacc7a86dd2d3d2a7 [id=eaf422968e464f1986f89b6944a6adbf]
I20260924 19:57:51.100620 19450 leader_election.cc:290] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689)
I20260924 19:57:51.102864 19398 raft_consensus.cc:493] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:51.103360 19398 raft_consensus.cc:515] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.105430 19398 leader_election.cc:290] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:51.107067 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78801b8d0b664217a74b20160ba2d8ac" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:51.107697 17819 raft_consensus.cc:2468] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:51.108879 17906 leader_election.cc:304] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:51.109648 19398 raft_consensus.cc:2804] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:57:51.109982 19398 raft_consensus.cc:493] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:51.110289 19398 raft_consensus.cc:3060] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:57:51.116123 18223 ts_tablet_manager.cc:1403] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.363s user 0.013s sys 0.003s
W20260924 19:57:51.079367 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 74207.3us
I20260924 19:57:51.087597 19204 raft_consensus.cc:515] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:51.117904 19398 raft_consensus.cc:515] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.118706 19204 leader_election.cc:290] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
W20260924 19:57:51.101366 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 48906.1us
I20260924 19:57:51.106402 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78801b8d0b664217a74b20160ba2d8ac" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:51.120100 18050 leader_election.cc:343] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.120944 18051 leader_election.cc:304] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:51.121671 19446 raft_consensus.cc:2804] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:51.122112 19446 raft_consensus.cc:697] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:51.122921 19446 consensus_queue.cc:237] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.124883 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78801b8d0b664217a74b20160ba2d8ac" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:51.124940 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78801b8d0b664217a74b20160ba2d8ac" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:51.125484 17819 raft_consensus.cc:3060] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:57:51.055078 19377 ts_tablet_manager.cc:1929] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:57:51.126183 19377 log.cc:1199] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/ae7a0660795e4e4b87d0959ca84d5d1b
I20260924 19:57:51.126868 19377 ts_tablet_manager.cc:1950] T ae7a0660795e4e4b87d0959ca84d5d1b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:57:51.129047 19377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 258) took 1662 ms. Trace:
I20260924 19:57:51.129382 19377 rpcz_store.cc:276] 0924 19:57:49.466935 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.467056 (+ 121us) service_pool.cc:224] Handling call
0924 19:57:50.305942 (+838886us) ts_tablet_manager.cc:1151] Deleting tablet ae7a0660795e4e4b87d0959ca84d5d1b
0924 19:57:50.305983 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:50.502882 (+196899us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:51.054949 (+552067us) tablet_metadata.cc:665] Metadata flushed
0924 19:57:51.128976 (+ 74027us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":838318}
I20260924 19:57:51.130575 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet ae7a0660795e4e4b87d0959ca84d5d1b (table default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df]) successfully deleted
I20260924 19:57:51.137091 17819 raft_consensus.cc:2468] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 2.
I20260924 19:57:51.138216 17906 leader_election.cc:304] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters: eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:51.138983 19433 raft_consensus.cc:2804] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:57:51.139474 19433 raft_consensus.cc:697] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 2 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:51.140334 19433 consensus_queue.cc:237] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.143769 18224 tablet_bootstrap.cc:492] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:51.147850 19377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 259) took 1679 ms. Trace:
I20260924 19:57:51.148241 19377 rpcz_store.cc:276] 0924 19:57:49.468242 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.468348 (+ 106us) service_pool.cc:224] Handling call
0924 19:57:51.147546 (+1679198us) ts_tablet_manager.cc:1151] Deleting tablet 201ea0020cb3430eac85e80b8c06fb07
0924 19:57:51.147599 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:51.147788 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1678713}
W20260924 19:57:51.149549 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 201ea0020cb3430eac85e80b8c06fb07 because the tablet was not found. No further retry: Not found: Tablet not found: 201ea0020cb3430eac85e80b8c06fb07
I20260924 19:57:51.149508 19398 leader_election.cc:290] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:51.155213 17819 raft_consensus.cc:1240] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 2 FOLLOWER]: Rejecting Update request from peer eaf64bfb64aa42948c9bc1411ce4ce35 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260924 19:57:51.166167 18224 tablet_bootstrap.cc:654] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:51.162284 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_8ade2ea47f344fd18a687f7d5e0ea568 [id=eecec62f4e6a4e56920d841de1b9077b]
I20260924 19:57:51.175976 19377 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38450 (request call id 257) took 1710 ms. Trace:
I20260924 19:57:51.176787 19377 rpcz_store.cc:276] 0924 19:57:49.465569 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.479683 (+ 14114us) service_pool.cc:224] Handling call
0924 19:57:51.175675 (+1695992us) ts_tablet_manager.cc:1151] Deleting tablet 07f86f24d9d14d05898e1c0f69428850
0924 19:57:51.175723 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:57:51.175914 (+ 191us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1695450}
W20260924 19:57:51.178165 17599 catalog_manager.cc:5030] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): delete failed for tablet 07f86f24d9d14d05898e1c0f69428850 because the tablet was not found. No further retry: Not found: Tablet not found: 07f86f24d9d14d05898e1c0f69428850
I20260924 19:57:51.180119 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfe0303596204e528585c4fe1c476d8b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:51.180701 17819 raft_consensus.cc:3060] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:51.170401 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_2bb6cac7492d4c5ca461d126788c9722 [id=f1f88aa92d884988b4bdf211754de8de]
I20260924 19:57:51.156306 18223 raft_consensus.cc:359] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.185662 18223 raft_consensus.cc:385] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:51.186288 19403 consensus_queue.cc:1059] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:51.187001 19403 raft_consensus.cc:3055] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Stepping down as leader of term 1
I20260924 19:57:51.187315 19403 raft_consensus.cc:740] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:51.188903 17970 raft_consensus.cc:1240] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [term 2 LEADER]: Rejecting Update request from peer eaf64bfb64aa42948c9bc1411ce4ce35 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260924 19:57:51.200450 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe]
I20260924 19:57:51.201890 17819 raft_consensus.cc:2468] T cfe0303596204e528585c4fe1c476d8b P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:51.204373 18220 tablet_bootstrap.cc:492] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:51.204829 18220 ts_tablet_manager.cc:1403] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.160s user 0.021s sys 0.009s
I20260924 19:57:51.207549 18220 raft_consensus.cc:359] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.208199 18220 raft_consensus.cc:385] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:51.208480 18220 raft_consensus.cc:740] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:51.209229 18220 consensus_queue.cc:260] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.217955 18220 ts_tablet_manager.cc:1434] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260924 19:57:51.218954 18220 tablet_bootstrap.cc:492] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:51.224649 18224 tablet_bootstrap.cc:492] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:51.225322 18224 ts_tablet_manager.cc:1403] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.004s
I20260924 19:57:51.227906 18224 raft_consensus.cc:359] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.228531 18224 raft_consensus.cc:385] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:51.228788 18224 raft_consensus.cc:740] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:51.228389 19408 consensus_queue.cc:1048] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260924 19:57:51.229440 18224 consensus_queue.cc:260] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.188019 19403 consensus_queue.cc:260] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.231341 19403 raft_consensus.cc:3060] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Advancing to term 2
W20260924 19:57:51.247561 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 51548.2us
I20260924 19:57:51.250874 18224 ts_tablet_manager.cc:1434] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260924 19:57:51.253522 18051 leader_election.cc:304] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:51.254281 19446 raft_consensus.cc:2804] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:51.185943 18223 raft_consensus.cc:740] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:51.257903 18223 consensus_queue.cc:260] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.260418 19446 raft_consensus.cc:697] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:51.262241 19402 raft_consensus.cc:493] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:51.264465 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfe0303596204e528585c4fe1c476d8b" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:51.264412 19402 raft_consensus.cc:515] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.266511 19402 leader_election.cc:290] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:51.267616 19446 consensus_queue.cc:237] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:51.273322 17967 raft_consensus.cc:3060] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:51.278751 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9f9b76a58e47c0beee5ce94a44b707" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:51.279377 17818 raft_consensus.cc:2468] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:51.285806 18939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:51.287187 18939 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ff67c5d57d474a9d92cc4c6de1287e in term 4.
I20260924 19:57:51.291867 17602 leader_election.cc:304] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e; no voters:
I20260924 19:57:51.292641 19450 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260924 19:57:51.292965 19450 raft_consensus.cc:493] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6f88e00b235f48bb82e6a3c9c9f51fa6)
I20260924 19:57:51.293282 19450 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 4 FOLLOWER]: Advancing to term 5
I20260924 19:57:51.300498 19450 raft_consensus.cc:515] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:51.302578 19450 leader_election.cc:290] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 5 election: Requested vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689)
I20260924 19:57:51.303354 18939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d"
I20260924 19:57:51.303875 18939 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 4 FOLLOWER]: Advancing to term 5
I20260924 19:57:51.304574 18223 ts_tablet_manager.cc:1434] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.165s user 0.005s sys 0.000s
I20260924 19:57:51.305503 18223 tablet_bootstrap.cc:492] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:51.306130 19436 raft_consensus.cc:493] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:51.306586 19436 raft_consensus.cc:515] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.308525 19436 leader_election.cc:290] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:51.309386 18051 leader_election.cc:304] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:51.310107 19204 raft_consensus.cc:2804] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:51.313911 19204 raft_consensus.cc:493] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:51.314234 19204 raft_consensus.cc:3060] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:51.314319 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_771765c449ce492986addb028e1b11d1 [id=f954ad85289641fda971e5c5155b9147]
I20260924 19:57:51.315843 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_b7e4d07630f24ee0ab58d5269687448d [id=f9f9606f84804c769a867198983fb713]
I20260924 19:57:51.317420 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_4e3a3158cde24302806b7ba7827831c4 [id=fa0b10df171340868b0305a28d192ea5]
I20260924 19:57:51.318751 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_88352246addd43f4be871c983413b2de [id=fa469a44fb3741d39036337129cc36a1]
I20260924 19:57:51.319583 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f116f866c2b4674b4eef531cf00af66" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:51.328054 18220 tablet_bootstrap.cc:654] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:51.332244 19336 raft_consensus.cc:493] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:51.332749 19336 raft_consensus.cc:515] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:51.335309 17753 leader_election.cc:343] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.336588 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9f9b76a58e47c0beee5ce94a44b707" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:51.337200 17818 raft_consensus.cc:2468] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:51.334896 19336 leader_election.cc:290] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:51.339936 17906 leader_election.cc:304] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:51.341073 19398 raft_consensus.cc:2804] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:51.341446 19398 raft_consensus.cc:493] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:51.341797 19398 raft_consensus.cc:3060] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:51.339973 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9f9b76a58e47c0beee5ce94a44b707" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:51.323139 19204 raft_consensus.cc:515] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.352592 18113 raft_consensus.cc:2393] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
W20260924 19:57:51.328156 18050 leader_election.cc:343] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.373317 18939 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ff67c5d57d474a9d92cc4c6de1287e in term 5.
I20260924 19:57:51.375519 18938 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Rejecting Update request from peer 6f88e00b235f48bb82e6a3c9c9f51fa6 for earlier term 4. Current term is 5. Ops: []
I20260924 19:57:51.379490 19362 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 }, Status: INVALID_TERM, Last received: 4.322, Next index: 323, Last known committed idx: 322, Time since last communication: 0.001s
I20260924 19:57:51.361920 19204 leader_election.cc:290] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:51.402958 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 75814.9us
I20260924 19:57:51.405879 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9f9b76a58e47c0beee5ce94a44b707" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:51.406471 17819 raft_consensus.cc:3060] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:51.407179 18220 tablet_bootstrap.cc:492] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:51.407743 18220 ts_tablet_manager.cc:1403] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.189s user 0.011s sys 0.002s
I20260924 19:57:51.414323 17819 raft_consensus.cc:2468] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:51.417317 18224 tablet_bootstrap.cc:492] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:51.424113 17602 leader_election.cc:304] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e; no voters:
I20260924 19:57:51.424863 19450 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 5 FOLLOWER]: Leader election won for term 5
I20260924 19:57:51.431892 19460 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 4 LEADER]: Stepping down as leader of term 4
I20260924 19:57:51.432370 19460 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6f88e00b235f48bb82e6a3c9c9f51fa6, State: Running, Role: LEADER
I20260924 19:57:51.433421 19460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 4.322, Last appended by leader: 322, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:51.436043 19460 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 4 FOLLOWER]: Advancing to term 5
W20260924 19:57:51.442670 17601 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet df722b0bbfd04ce2943f0e5b8058c50f (table default.Table_db4a5e39fd954bb0b4c271899feefe90 [id=0b7e96c71a074f838a519746d9c92c3f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet df722b0bbfd04ce2943f0e5b8058c50f consensus state has no leader
I20260924 19:57:51.443724 18223 tablet_bootstrap.cc:654] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:51.443606 19398 raft_consensus.cc:515] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.448201 19450 raft_consensus.cc:697] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [term 5 LEADER]: Becoming Leader. State: Replica: e1ff67c5d57d474a9d92cc4c6de1287e, State: Running, Role: LEADER
W20260924 19:57:51.449314 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 45722.4us
I20260924 19:57:51.449270 19450 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 4.322, Last appended by leader: 322, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:51.450747 19374 consensus_queue.cc:1048] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:51.458043 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 54426us
I20260924 19:57:51.460862 18051 leader_election.cc:304] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:51.461678 19403 raft_consensus.cc:2804] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:51.435056 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_2255ca45bfcf4f819376770df3186e24 [id=fb5bd2d87d78475d803e753473de0c7a]
I20260924 19:57:51.465582 19403 raft_consensus.cc:697] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:51.465749 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_ee93a1803fd54a4ba6723d1e3783c004 [id=fec7f8aa891e42c1a84fe32fe4e0c44f]
I20260924 19:57:51.466435 19403 consensus_queue.cc:237] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.467113 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_c449d7c2b02a4bcdb7091a3560bcaa02 [id=ff6c3885d44548b78bb3e4c556a5f383]
I20260924 19:57:51.468488 19297 catalog_manager.cc:692] Loaded metadata for table default.Table_a36de9161f8f4ea08243a7222c099822 [id=ff9a4ba6b88648748c228135ed440066]
I20260924 19:57:51.460945 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9f9b76a58e47c0beee5ce94a44b707" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:51.410952 18220 raft_consensus.cc:359] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.464134 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9f9b76a58e47c0beee5ce94a44b707" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:51.472859 18220 raft_consensus.cc:385] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:51.483465 18223 tablet_bootstrap.cc:492] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:51.483915 18223 ts_tablet_manager.cc:1403] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.179s user 0.008s sys 0.003s
I20260924 19:57:51.485932 19461 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ff67c5d57d474a9d92cc4c6de1287e. Latest consensus state: current_term: 5 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:51.486421 18223 raft_consensus.cc:359] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.473369 18220 raft_consensus.cc:740] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:51.489527 19336 raft_consensus.cc:493] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:51.495580 19336 raft_consensus.cc:515] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.497669 19336 leader_election.cc:290] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:51.498359 18107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96379e7356ab40218c6ee2c126c0b4ae" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:51.499681 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96379e7356ab40218c6ee2c126c0b4ae" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:51.501233 17906 leader_election.cc:304] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:51.486891 19461 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:51.502519 19336 raft_consensus.cc:2749] T 96379e7356ab40218c6ee2c126c0b4ae P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:51.502624 19403 consensus_queue.cc:1048] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:51.506510 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f116f866c2b4674b4eef531cf00af66" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:51.509338 19398 leader_election.cc:290] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:51.511485 17970 raft_consensus.cc:2393] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eaf64bfb64aa42948c9bc1411ce4ce35 in current term 1: Already voted for candidate f74e6ed351284d7fb110e44336ce643a in this term.
I20260924 19:57:51.512444 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9f9b76a58e47c0beee5ce94a44b707" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:51.512741 18220 consensus_queue.cc:260] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.519570 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9f9b76a58e47c0beee5ce94a44b707" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:51.520498 17819 raft_consensus.cc:2393] T 5e9f9b76a58e47c0beee5ce94a44b707 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:51.521044 18224 tablet_bootstrap.cc:654] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:51.521775 17906 leader_election.cc:304] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:51.522910 19336 raft_consensus.cc:2749] T 5e9f9b76a58e47c0beee5ce94a44b707 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 19:57:51.531689 18220 ts_tablet_manager.cc:1434] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.124s user 0.007s sys 0.000s
W20260924 19:57:51.541172 17753 leader_election.cc:343] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.543709 17753 leader_election.cc:304] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:51.547245 19436 raft_consensus.cc:2749] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:51.487013 18223 raft_consensus.cc:385] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:51.549664 18223 raft_consensus.cc:740] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:51.550546 18223 consensus_queue.cc:260] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.562562 18220 tablet_bootstrap.cc:492] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:51.569249 18220 tablet_bootstrap.cc:654] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:51.582751 18224 tablet_bootstrap.cc:492] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:51.583228 18224 ts_tablet_manager.cc:1403] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.166s user 0.011s sys 0.000s
I20260924 19:57:51.619506 18224 raft_consensus.cc:359] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.620188 18224 raft_consensus.cc:385] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:51.620505 18224 raft_consensus.cc:740] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:51.633117 18223 ts_tablet_manager.cc:1434] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.149s user 0.005s sys 0.000s
W20260924 19:57:51.641481 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 44192us
I20260924 19:57:51.642515 19374 raft_consensus.cc:493] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:51.642976 19374 raft_consensus.cc:515] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.645059 19374 leader_election.cc:290] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:51.648501 18223 tablet_bootstrap.cc:492] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:51.654271 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335f979eb04c49bcb53023693f54ad41" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:51.654950 17818 raft_consensus.cc:2468] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:51.658471 18051 leader_election.cc:304] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:51.661319 19374 raft_consensus.cc:2804] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:51.662679 19374 raft_consensus.cc:493] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:51.663123 19374 raft_consensus.cc:3060] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:51.663699 19467 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260924 19:57:51.665429 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335f979eb04c49bcb53023693f54ad41" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:51.677407 18050 leader_election.cc:343] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.682852 18223 tablet_bootstrap.cc:654] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:51.687069 18220 tablet_bootstrap.cc:492] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:51.687539 18220 ts_tablet_manager.cc:1403] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.125s user 0.007s sys 0.004s
I20260924 19:57:51.688973 19374 raft_consensus.cc:515] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.691049 19374 leader_election.cc:290] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:51.692157 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335f979eb04c49bcb53023693f54ad41" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:51.692714 17818 raft_consensus.cc:3060] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:51.696743 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335f979eb04c49bcb53023693f54ad41" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:51.697754 19436 raft_consensus.cc:493] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:51.698191 19436 raft_consensus.cc:515] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.700018 19436 leader_election.cc:290] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:51.704104 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7a1887d396f40a1a0c153dd0b2fe307" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:51.705312 17753 leader_election.cc:343] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.717010 18224 consensus_queue.cc:260] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.717875 19436 raft_consensus.cc:493] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:51.718338 19436 raft_consensus.cc:515] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.723625 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7a1887d396f40a1a0c153dd0b2fe307" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:51.724756 18050 leader_election.cc:343] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:51.726073 17753 leader_election.cc:343] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.726459 17753 leader_election.cc:304] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:51.731905 18220 raft_consensus.cc:359] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.732653 18220 raft_consensus.cc:385] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:51.732942 18220 raft_consensus.cc:740] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:51.733271 19436 leader_election.cc:290] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:51.733649 18220 consensus_queue.cc:260] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.735416 18220 ts_tablet_manager.cc:1434] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.048s user 0.005s sys 0.001s
I20260924 19:57:51.737238 19436 raft_consensus.cc:2749] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:51.739006 19446 consensus_queue.cc:1048] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:51.740818 17818 raft_consensus.cc:2468] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:51.744582 18051 leader_election.cc:304] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:51.745298 19446 raft_consensus.cc:2804] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:51.745744 19446 raft_consensus.cc:697] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:51.746533 19446 consensus_queue.cc:237] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.761665 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf025253c49a4c23b79a25333cad160e" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:51.763257 17753 leader_election.cc:343] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.799436 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf025253c49a4c23b79a25333cad160e" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:51.800741 17753 leader_election.cc:343] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.801213 17753 leader_election.cc:304] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:51.802445 19398 raft_consensus.cc:493] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:51.802879 19398 raft_consensus.cc:515] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.804823 19398 leader_election.cc:290] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:51.807924 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fd0d25961d74b67a42323a045f06b61" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:51.808477 18113 raft_consensus.cc:2468] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:51.810001 17905 leader_election.cc:304] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:51.810964 18220 tablet_bootstrap.cc:492] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:51.812402 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fd0d25961d74b67a42323a045f06b61" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:51.813001 17818 raft_consensus.cc:2468] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:51.815047 18223 tablet_bootstrap.cc:492] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:51.815477 18223 ts_tablet_manager.cc:1403] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.167s user 0.010s sys 0.000s
I20260924 19:57:51.811316 19433 raft_consensus.cc:2804] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:51.816008 19433 raft_consensus.cc:493] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:51.816335 19433 raft_consensus.cc:3060] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:51.825853 19475 raft_consensus.cc:2749] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:51.826841 18224 ts_tablet_manager.cc:1434] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.243s user 0.009s sys 0.000s
I20260924 19:57:51.828032 18224 tablet_bootstrap.cc:492] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:51.839064 18939 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Refusing update from remote peer e1ff67c5d57d474a9d92cc4c6de1287e: Log matching property violated. Preceding OpId in replica: term: 4 index: 322. Preceding OpId from leader: term: 5 index: 323. (index mismatch)
I20260924 19:57:51.841167 19461 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20260924 19:57:51.872661 19472 raft_consensus.cc:493] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:51.873214 19472 raft_consensus.cc:515] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.875886 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7635c52883ec4434bad0cc15f8733f72" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:51.877241 17753 leader_election.cc:343] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.877288 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7635c52883ec4434bad0cc15f8733f72" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:51.878252 19472 leader_election.cc:290] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:51.879617 19460 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:57:51.878780 17753 leader_election.cc:343] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:51.880119 19460 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:51.880473 17753 leader_election.cc:304] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:51.881119 19474 raft_consensus.cc:2749] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:51.883517 19460 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:51.884019 18938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
W20260924 19:57:51.885072 18899 leader_election.cc:336] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689): Timed out: RequestConsensusVote RPC to 127.17.2.189:43689 timed out after 0.528s (SENT)
I20260924 19:57:51.885619 18899 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:51.846691 19433 raft_consensus.cc:515] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.899778 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fd0d25961d74b67a42323a045f06b61" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:51.829509 18223 raft_consensus.cc:359] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.900277 18113 raft_consensus.cc:3060] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:51.900689 18223 raft_consensus.cc:385] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:51.906617 18113 raft_consensus.cc:2468] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:51.908739 19336 raft_consensus.cc:493] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:51.909214 19336 raft_consensus.cc:515] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.910367 17905 leader_election.cc:304] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:51.911029 19398 raft_consensus.cc:2804] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:51.913046 19336 leader_election.cc:290] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:51.917598 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335f979eb04c49bcb53023693f54ad41" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:51.917536 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "335f979eb04c49bcb53023693f54ad41" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:51.926419 17818 raft_consensus.cc:2393] T 335f979eb04c49bcb53023693f54ad41 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:51.927618 17906 leader_election.cc:304] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:51.928794 19336 raft_consensus.cc:3060] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:51.903622 18223 raft_consensus.cc:740] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:51.938412 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6fd0d25961d74b67a42323a045f06b61" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:51.939087 17818 raft_consensus.cc:3060] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:51.941048 18899 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.189:43689, user_credentials={real_user=slave}} blocked reactor thread for 56106.2us
I20260924 19:57:51.943778 19398 raft_consensus.cc:697] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:51.946841 19398 consensus_queue.cc:237] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.947904 18223 consensus_queue.cc:260] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:51.949015 19433 leader_election.cc:290] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:51.965819 18224 tablet_bootstrap.cc:654] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:51.989105 18223 ts_tablet_manager.cc:1434] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.159s user 0.006s sys 0.000s
I20260924 19:57:51.982998 19336 raft_consensus.cc:2749] T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:51.953835 18220 tablet_bootstrap.cc:654] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:52.022305 19297 tablet_loader.cc:96] loaded metadata for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82])
I20260924 19:57:52.013976 17818 raft_consensus.cc:2468] T 6fd0d25961d74b67a42323a045f06b61 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:52.040046 18223 tablet_bootstrap.cc:492] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:52.071733 18224 tablet_bootstrap.cc:492] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:52.072221 18224 ts_tablet_manager.cc:1403] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.244s user 0.012s sys 0.000s
I20260924 19:57:52.074995 18224 raft_consensus.cc:359] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:52.094551 19483 log.cc:927] Time spent T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a: Append to log took a long time: real 0.104s user 0.000s sys 0.002s
I20260924 19:57:52.102094 18113 raft_consensus.cc:1275] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 2 FOLLOWER]: Refusing update from remote peer f74e6ed351284d7fb110e44336ce643a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260924 19:57:52.102422 18113 pending_rounds.cc:85] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35: Aborting all ops after (but not including) 0
I20260924 19:57:52.102662 18113 pending_rounds.cc:107] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260924 19:57:52.102902 18113 raft_consensus.cc:2889] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260924 19:57:52.104398 19482 consensus_queue.cc:1048] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:52.083642 18224 raft_consensus.cc:385] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:52.114687 19478 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260924 19:57:52.110236 18224 raft_consensus.cc:740] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:52.118624 19479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ff67c5d57d474a9d92cc4c6de1287e. Latest consensus state: current_term: 5 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:52.123744 19479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:52.144716 19472 raft_consensus.cc:493] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:52.145183 19472 raft_consensus.cc:515] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.164376 18224 consensus_queue.cc:260] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.166692 17819 raft_consensus.cc:1275] T 78801b8d0b664217a74b20160ba2d8ac P c4f6e3a7b06148918d230c77ab0c8202 [term 2 FOLLOWER]: Refusing update from remote peer f74e6ed351284d7fb110e44336ce643a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260924 19:57:52.176631 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accddc7f696142bfa6a5e655f5abc220" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:52.177451 18220 tablet_bootstrap.cc:492] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:52.177837 18220 ts_tablet_manager.cc:1403] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.367s user 0.006s sys 0.010s
W20260924 19:57:52.178002 17753 leader_election.cc:343] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:52.179821 18107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accddc7f696142bfa6a5e655f5abc220" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:52.180500 19336 consensus_queue.cc:1048] T 78801b8d0b664217a74b20160ba2d8ac P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:52.181712 17753 leader_election.cc:343] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:52.182098 17753 leader_election.cc:304] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:52.179349 19472 leader_election.cc:290] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:52.184041 19475 raft_consensus.cc:2749] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:52.185942 18224 ts_tablet_manager.cc:1434] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.113s user 0.006s sys 0.000s
I20260924 19:57:52.186605 18223 tablet_bootstrap.cc:654] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:52.200001 19297 tablet_loader.cc:96] loaded metadata for tablet 0b99ff1920274831b58d8283bd592818 (table default.Table_aeac9df2b26b490a83f10d672335d75e [id=80222cdc26da492e9f386e9eb2fb77b8])
W20260924 19:57:52.206552 18050 consensus_peers.cc:597] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:57:52.208685 19475 raft_consensus.cc:493] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:52.209128 19475 raft_consensus.cc:515] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.211282 19475 leader_election.cc:290] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:52.211961 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25dd98712d54cd5b53df00b2cf9f3c4" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:52.212549 17972 raft_consensus.cc:2468] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:52.214877 17753 leader_election.cc:304] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:52.216154 18107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25dd98712d54cd5b53df00b2cf9f3c4" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:52.216920 18107 raft_consensus.cc:2468] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:52.219064 19475 raft_consensus.cc:2804] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:52.219604 19475 raft_consensus.cc:493] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:52.220124 19475 raft_consensus.cc:3060] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:52.230732 18220 raft_consensus.cc:359] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.231426 18220 raft_consensus.cc:385] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:52.231662 18220 raft_consensus.cc:740] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:52.232218 18220 consensus_queue.cc:260] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.233716 18220 ts_tablet_manager.cc:1434] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.056s user 0.004s sys 0.001s
I20260924 19:57:52.257766 19297 tablet_loader.cc:96] loaded metadata for tablet 0c08f10d2a364cb8a0a7ea40654c3fe4 (table default.Table_2bb6cac7492d4c5ca461d126788c9722 [id=f1f88aa92d884988b4bdf211754de8de])
I20260924 19:57:52.259673 19297 tablet_loader.cc:96] loaded metadata for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_93373f379bb14b6ca7a3ac7a628cad63 [id=5878a9a1b1d54055a5e8dc712b9de665])
W20260924 19:57:52.263478 17599 leader_election.cc:336] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689): Timed out: RequestConsensusVote RPC to 127.17.2.189:43689 timed out after 0.574s (SENT)
I20260924 19:57:52.274326 19478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:52.275041 19478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:52.283007 19461 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:52.283792 19461 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:52.292708 18220 tablet_bootstrap.cc:492] T cfe0303596204e528585c4fe1c476d8b P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:52.296118 19297 tablet_loader.cc:96] loaded metadata for tablet 0cf2df461598411f8ad9f3a51eafc6f7 (table default.Table_a10031a0fa004ac3ad48e6020477494b [id=7aab4014a5bc4823bbe845c021153fc9])
I20260924 19:57:52.298146 19297 tablet_loader.cc:96] loaded metadata for tablet 10fe8d2d06dd4f83bef8c52a82693344 (table default.Table_fcbe3b81791a4aaf923847b1366eb6f9 [id=25cb87a38f0f473d99bad6f06d4edcf4])
I20260924 19:57:52.317464 18224 tablet_bootstrap.cc:492] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:52.332377 19475 raft_consensus.cc:515] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.340822 19297 tablet_loader.cc:96] loaded metadata for tablet 144355b18c2f413baa28dd632623db52 (table default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b [id=7babe5418e104e76871835ca3e30f95c])
I20260924 19:57:52.354132 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25dd98712d54cd5b53df00b2cf9f3c4" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:52.354852 17970 raft_consensus.cc:3060] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:52.363580 17970 raft_consensus.cc:2468] T b25dd98712d54cd5b53df00b2cf9f3c4 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:52.366693 17753 leader_election.cc:304] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:52.367420 18220 tablet_bootstrap.cc:654] T cfe0303596204e528585c4fe1c476d8b P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:52.367722 19472 raft_consensus.cc:2804] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:52.363468 19297 tablet_loader.cc:96] loaded metadata for tablet 18bb8f3d43df4aae8614b6dba9ffc668 (table default.Table_2255ca45bfcf4f819376770df3186e24 [id=fb5bd2d87d78475d803e753473de0c7a])
I20260924 19:57:52.368547 19472 raft_consensus.cc:697] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:52.369390 19472 consensus_queue.cc:237] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.374837 19475 leader_election.cc:290] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:52.384871 18224 tablet_bootstrap.cc:654] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:52.399546 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25dd98712d54cd5b53df00b2cf9f3c4" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:52.400195 18112 raft_consensus.cc:3060] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:52.411437 17521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.527s (SENT)
I20260924 19:57:52.419145 18220 tablet_bootstrap.cc:492] T cfe0303596204e528585c4fe1c476d8b P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:52.419622 18220 ts_tablet_manager.cc:1403] T cfe0303596204e528585c4fe1c476d8b P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.127s user 0.012s sys 0.000s
I20260924 19:57:52.423573 18220 raft_consensus.cc:359] T cfe0303596204e528585c4fe1c476d8b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:52.425346 18220 raft_consensus.cc:385] T cfe0303596204e528585c4fe1c476d8b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:52.425633 18220 raft_consensus.cc:740] T cfe0303596204e528585c4fe1c476d8b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:52.427065 18220 consensus_queue.cc:260] T cfe0303596204e528585c4fe1c476d8b P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:52.430912 18220 ts_tablet_manager.cc:1434] T cfe0303596204e528585c4fe1c476d8b P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
W20260924 19:57:52.444414 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 145659us
I20260924 19:57:52.450177 18223 tablet_bootstrap.cc:492] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:52.450603 18223 ts_tablet_manager.cc:1403] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.415s user 0.016s sys 0.006s
I20260924 19:57:52.454180 19297 tablet_loader.cc:96] loaded metadata for tablet 274f3decdc7e42c0917bf28151c4da71 (table default.Table_954c770a550c49599e38ee3850e4aea1 [id=c4ad22b552204e2d8209b275e57eb9c4])
I20260924 19:57:52.466795 17970 raft_consensus.cc:3060] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:52.451514 18112 raft_consensus.cc:2468] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:52.486760 19297 tablet_loader.cc:96] loaded metadata for tablet 28fb29faeeb34c57bb6e52a9c5ffd218 (table default.Table_843b88f433f044458daa98228935a46e [id=c19508f168ed4191bdd25de1107ab416])
I20260924 19:57:52.488243 19297 tablet_loader.cc:96] loaded metadata for tablet 2a1ac7e1b192465696640c02898bc39f (table default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b])
I20260924 19:57:52.489537 19297 tablet_loader.cc:96] loaded metadata for tablet 2b0b011357674a42a09d83f9384abaf1 (table default.Table_5aeda90fd8e3425a8c49a239533690c9 [id=63622937f5a64a05b56aa1310407ec99])
I20260924 19:57:52.494109 18220 tablet_bootstrap.cc:492] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:52.495901 19503 consensus_queue.cc:1048] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:52.500288 18220 tablet_bootstrap.cc:654] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:52.506313 18223 raft_consensus.cc:359] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:52.507117 18223 raft_consensus.cc:385] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:52.507448 18223 raft_consensus.cc:740] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:52.528869 18223 consensus_queue.cc:260] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:52.539829 18224 tablet_bootstrap.cc:492] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:52.540251 18224 ts_tablet_manager.cc:1403] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.223s user 0.019s sys 0.026s
I20260924 19:57:52.542567 18224 raft_consensus.cc:359] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.543218 18224 raft_consensus.cc:385] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:52.543478 18224 raft_consensus.cc:740] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:52.540311 19498 consensus_queue.cc:1048] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:52.544155 18224 consensus_queue.cc:260] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.546679 18223 ts_tablet_manager.cc:1434] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.096s user 0.006s sys 0.000s
I20260924 19:57:52.547685 18223 tablet_bootstrap.cc:492] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:52.553471 18224 ts_tablet_manager.cc:1434] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260924 19:57:52.554414 18224 tablet_bootstrap.cc:492] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:52.561544 19375 consensus_queue.cc:1048] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:52.562551 18220 tablet_bootstrap.cc:492] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:52.562950 18220 ts_tablet_manager.cc:1403] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.000s
I20260924 19:57:52.564229 19375 consensus_queue.cc:1048] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:52.565601 18220 raft_consensus.cc:359] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.566296 18220 raft_consensus.cc:385] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:52.566565 18220 raft_consensus.cc:740] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:52.590641 18220 consensus_queue.cc:260] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.592710 18220 ts_tablet_manager.cc:1434] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20260924 19:57:52.593528 18220 tablet_bootstrap.cc:492] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:52.601051 19297 tablet_loader.cc:96] loaded metadata for tablet 2de3877b94b44c94b6d0fdad93f8aa79 (table default.Table_65d20bb187ea4069b2e589e7643028be [id=00e4a8cdcb374ae6a4446b3b7404d7a8])
I20260924 19:57:52.603003 19297 tablet_loader.cc:96] loaded metadata for tablet 2e6d926ff7c24901a15b0c7b2239fbad (table default.Table_7c27cdce0a3b40f69142de52aaf39493 [id=65f397bcbc164d0caa6cecae1bab74da])
I20260924 19:57:52.629822 19297 tablet_loader.cc:96] loaded metadata for tablet 335f979eb04c49bcb53023693f54ad41 (table default.Table_ae1050d7716043fbad49a8f62505e719 [id=8681c352f02242a69511794ee585bf28])
I20260924 19:57:52.641645 19297 tablet_loader.cc:96] loaded metadata for tablet 3b328d37d55e4116a29322caaa21d431 (table default.Table_4e897947eb9e4466a9f050091b6a7b60 [id=b7032f43d384487d9b041986780669c0])
I20260924 19:57:52.637769 18220 tablet_bootstrap.cc:654] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:52.650863 18223 tablet_bootstrap.cc:654] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:52.653595 19297 tablet_loader.cc:96] loaded metadata for tablet 3d25793b65da4315b6f95726374ea46c (table default.Table_5cc282e71cd74150a849aff1620d54f1 [id=010f7a5a46034f008e83a6492524b778])
I20260924 19:57:52.654700 19297 tablet_loader.cc:96] loaded metadata for tablet 3e52ecf08c6c45d4b8ba423358fb5fc1 (table default.Table_dea06619a5364df0961e47cb617c9e52 [id=e40a55e03a2d4ceea51d555d0c60a334])
I20260924 19:57:52.650888 19474 raft_consensus.cc:493] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:52.655911 19474 raft_consensus.cc:515] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:52.657893 19474 leader_election.cc:290] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:52.662169 19297 tablet_loader.cc:96] loaded metadata for tablet 3eb3d893415b4dfcbcacd435c3c16e5e (table default.Table_c42c4d9b6c1e4d2abd6aa51f251be773 [id=a72d39e753b2435a800e68d308cdc86c])
I20260924 19:57:52.671590 19336 raft_consensus.cc:493] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:52.672098 19336 raft_consensus.cc:515] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.673710 19297 tablet_loader.cc:96] loaded metadata for tablet 431d298f513d47268284d50af2ea1436 (table default.Table_6b68af48dd4140dda7d1854a3cf2269f [id=1a5ed525f4364782a0793571d4caa5cb])
I20260924 19:57:52.674198 19336 leader_election.cc:290] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:52.676393 19297 tablet_loader.cc:96] loaded metadata for tablet 46e703a4892f445bb0a1c9f5a46db63d (table default.Table_ee93a1803fd54a4ba6723d1e3783c004 [id=fec7f8aa891e42c1a84fe32fe4e0c44f])
I20260924 19:57:52.676242 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d930e7f4a6b7471899dab1b28e40f93d" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:52.677629 19297 tablet_loader.cc:96] loaded metadata for tablet 484517ee7df8463bb70d201ead1c6835 (table default.Table_771765c449ce492986addb028e1b11d1 [id=f954ad85289641fda971e5c5155b9147])
I20260924 19:57:52.678553 18224 tablet_bootstrap.cc:654] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:52.678648 19297 tablet_loader.cc:96] loaded metadata for tablet 492eb083f50e40a388d2174d6dc8445b (table default.Table_9faf28b24a2c47449a862536ea873f36 [id=3d02208bd3604c729b474a4f005b8cf8])
I20260924 19:57:52.680881 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d930e7f4a6b7471899dab1b28e40f93d" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:52.682157 18223 tablet_bootstrap.cc:492] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:52.682591 18223 ts_tablet_manager.cc:1403] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.135s user 0.010s sys 0.000s
I20260924 19:57:52.683270 19478 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:52.683749 19478 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:52.686478 19478 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:52.689963 18223 raft_consensus.cc:359] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.693015 19474 raft_consensus.cc:493] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:52.693554 19474 raft_consensus.cc:515] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:52.690678 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 130532us
I20260924 19:57:52.695611 19474 leader_election.cc:290] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:52.695782 17905 leader_election.cc:304] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:52.699759 19482 raft_consensus.cc:2749] T d930e7f4a6b7471899dab1b28e40f93d P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:52.700802 18223 raft_consensus.cc:385] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:52.702028 18223 raft_consensus.cc:740] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:52.702741 18223 consensus_queue.cc:260] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.707020 18223 ts_tablet_manager.cc:1434] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20260924 19:57:52.720088 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a1ac7e1b192465696640c02898bc39f" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:52.724644 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a1ac7e1b192465696640c02898bc39f" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:52.725289 18112 raft_consensus.cc:2468] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:52.726002 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb111f7a189c4f00a41686d647c64e65" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:52.727982 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb111f7a189c4f00a41686d647c64e65" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:52.728343 19297 tablet_loader.cc:96] loaded metadata for tablet 56295c0109ce489fb03d65674df62c44 (table default.Table_b7e4d07630f24ee0ab58d5269687448d [id=f9f9606f84804c769a867198983fb713])
I20260924 19:57:52.730384 19297 tablet_loader.cc:96] loaded metadata for tablet 57102b1727e94a90981509c82c082441 (table default.Table_2a8be69ab9604081a07df5197e7fab9d [id=8d382cdb38764b11ba323546fff603bf])
I20260924 19:57:52.732959 18223 tablet_bootstrap.cc:492] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:52.741271 19336 consensus_queue.cc:1048] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:52.768344 19297 tablet_loader.cc:96] loaded metadata for tablet 5df3dcf4ddcf4efa9aa047da56dfb7a7 (table default.Table_a82c256e36e448c296c86a30176f4135 [id=757ccfb2d39f4907a2580b365d283ace])
I20260924 19:57:52.770049 19297 tablet_loader.cc:96] loaded metadata for tablet 5e9f9b76a58e47c0beee5ce94a44b707 (table default.Table_c449d7c2b02a4bcdb7091a3560bcaa02 [id=ff6c3885d44548b78bb3e4c556a5f383])
I20260924 19:57:52.778997 19474 raft_consensus.cc:493] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:52.779491 19474 raft_consensus.cc:515] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:52.781513 19474 leader_election.cc:290] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:52.787952 19297 tablet_loader.cc:96] loaded metadata for tablet 66599c38b1a64ad28fe7bd8e5b5056a8 (table default.Table_a50ccb6bd6a042bc99b503e5aa1188ad [id=bfd5e6e55fb049f9965b190374b38c3d])
W20260924 19:57:52.794535 17753 leader_election.cc:343] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:52.806341 19297 tablet_loader.cc:96] loaded metadata for tablet 6b7f2dd8ba2a4cb190e9ff92a542bacb (table default.Table_11ca95d108694c9498d4de67c9ff8808 [id=bec51c14454046329b3eb7016389f235])
I20260924 19:57:52.809834 19474 raft_consensus.cc:493] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:52.810664 19297 tablet_loader.cc:96] loaded metadata for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f])
I20260924 19:57:52.810309 19474 raft_consensus.cc:515] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.812398 19474 leader_election.cc:290] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:52.813334 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c1afbfc7de7480ab4be40680a3e3ada" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:52.821157 19297 tablet_loader.cc:96] loaded metadata for tablet 6fd0d25961d74b67a42323a045f06b61 (table default.Table_fbc5ceeb4d1242d19344825cefb6ef2a [id=caf8f776b4754b80a6aac8211f1b7a6f])
I20260924 19:57:52.823880 19297 tablet_loader.cc:96] loaded metadata for tablet 70fea649583648ca86b4b3673ab0305f (table default.Table_66edcc97f7a64dfa85bf24d26e49c5de [id=31d1abb2c65744d78c5e61dd4ae81b63])
I20260924 19:57:52.832413 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c1afbfc7de7480ab4be40680a3e3ada" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:52.823298 19510 consensus_queue.cc:1048] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260924 19:57:52.833052 19482 consensus_queue.cc:1048] T 6fd0d25961d74b67a42323a045f06b61 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:52.835222 18223 tablet_bootstrap.cc:654] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:52.846730 17599 leader_election.cc:336] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689): Timed out: RequestConsensusVote RPC to 127.17.2.189:43689 timed out after 0.707s (SENT)
I20260924 19:57:52.849085 19297 tablet_loader.cc:96] loaded metadata for tablet 7635c52883ec4434bad0cc15f8733f72 (table default.Table_0673b53c69bf4c429697ba739ef113e4 [id=ea252b103a0f43d18971c78fe2174c2e])
I20260924 19:57:52.851583 19297 tablet_loader.cc:96] loaded metadata for tablet 78801b8d0b664217a74b20160ba2d8ac (table default.Table_454683a3d79d4988b41292f7776df549 [id=81b26242949442c0a7db6f55f0f18097])
I20260924 19:57:52.858289 19297 tablet_loader.cc:96] loaded metadata for tablet 7ba365c977984e68a316356465b1e36a (table default.Table_05427967e4cc480cba4d8f7128b23677 [id=11bd6526602f4c0dbdb90bbc8806262b])
I20260924 19:57:52.860761 19408 consensus_queue.cc:1048] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:52.873730 19297 tablet_loader.cc:96] loaded metadata for tablet 7c1afbfc7de7480ab4be40680a3e3ada (table default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027])
I20260924 19:57:52.892134 19297 tablet_loader.cc:96] loaded metadata for tablet 7c7b8422753a4746bde71fcb25409949 (table default.Table_388c160758d44e1e89b8cb5d541316f8 [id=70f30a75b8d443a0ae28243adf40862a])
I20260924 19:57:52.894140 18220 tablet_bootstrap.cc:492] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:52.894754 18220 ts_tablet_manager.cc:1403] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.301s user 0.025s sys 0.014s
I20260924 19:57:52.899266 19297 tablet_loader.cc:96] loaded metadata for tablet 83cfc9998eb84e66b61e88b010992e3e (table default.Table_0c6b7fd4e7bd42a3ba0d407aea438273 [id=79e6d48b52b740958363f06cf05f2f63])
I20260924 19:57:52.899372 19498 raft_consensus.cc:493] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:52.899648 18220 raft_consensus.cc:359] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.899856 19498 raft_consensus.cc:515] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.902642 19297 tablet_loader.cc:96] loaded metadata for tablet 8523125e35554830aff04b17677d5ceb (table default.Table_5e789131162643caa4c014a55244066f [id=52ff52bfd3904ab0b4826673127f70ac])
I20260924 19:57:52.906170 18220 raft_consensus.cc:385] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:52.906245 19498 leader_election.cc:290] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:52.906683 18220 raft_consensus.cc:740] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:52.907621 18220 consensus_queue.cc:260] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.853636 18224 tablet_bootstrap.cc:492] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:52.908825 18224 ts_tablet_manager.cc:1403] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.355s user 0.008s sys 0.003s
I20260924 19:57:52.910712 18220 ts_tablet_manager.cc:1434] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260924 19:57:52.911171 19297 tablet_loader.cc:96] loaded metadata for tablet 872bc32b9e384c828a0699079b27530d (table default.Table_3bf03540f35b468dacc7a86dd2d3d2a7 [id=eaf422968e464f1986f89b6944a6adbf])
I20260924 19:57:52.912446 19297 tablet_loader.cc:96] loaded metadata for tablet 89646893588340038f30ed4754623ec2 (table default.Table_bb9775fce761443ca3a306e0cace741e [id=dc2b2e2afb22446ba04263f4f86a23e0])
I20260924 19:57:52.816519 17753 leader_election.cc:304] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:52.918885 19474 raft_consensus.cc:2804] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260924 19:57:52.918991 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 102714us
I20260924 19:57:52.919291 19474 raft_consensus.cc:493] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:52.919660 19474 raft_consensus.cc:3060] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:52.919698 17753 leader_election.cc:343] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:52.920324 17753 leader_election.cc:343] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:52.922142 19297 tablet_loader.cc:96] loaded metadata for tablet 8c05ec2ba01448d998c6ffa1525de984 (table default.Table_14bf9267f2da4ba48354027fc92cab3d [id=b898f46c59b246e39fe8b62a45019ecb])
I20260924 19:57:52.923218 19297 tablet_loader.cc:96] loaded metadata for tablet 8ea2385391d34f8abd9d105fbd7f22cf (table default.Table_a36de9161f8f4ea08243a7222c099822 [id=ff9a4ba6b88648748c228135ed440066])
I20260924 19:57:52.936242 19297 tablet_loader.cc:96] loaded metadata for tablet 924bc897f48b44069f1fb2ee71e26db7 (table default.Table_415dca00ed274e59961ddf45e42de178 [id=b8a8b30617164fbcb9f6918142768362])
I20260924 19:57:52.937508 19297 tablet_loader.cc:96] loaded metadata for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_88b623b6a00845afb22e24e34604e9ea [id=72c7d8cc98ee49679d8c7589a9ac0c8f])
I20260924 19:57:52.938716 19297 tablet_loader.cc:96] loaded metadata for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_7b49c63ad79c4132afbbbb9e1f2d4ca0 [id=8edea4510fd84d40819b48b401ccfe64])
I20260924 19:57:52.943648 19297 tablet_loader.cc:96] loaded metadata for tablet 96379e7356ab40218c6ee2c126c0b4ae (table default.Table_39ac82df7ce14073912622fcd1b57aaa [id=1ae05845cc1641ffaf3b82b0f1390e43])
I20260924 19:57:52.945597 19297 tablet_loader.cc:96] loaded metadata for tablet 97bbb17c170d42349d2c9cdb41349cb9 (table default.Table_b2824cbbb81c4186840b758c61eccbb2 [id=4e169b185baf4e7a81735a13b6d82fca])
I20260924 19:57:52.946580 19297 tablet_loader.cc:96] loaded metadata for tablet 9936ffa261ea4aa98426b00a4ef3c6a9 (table default.Table_b6d3fa192eb741158adf5b69323a29de [id=9ead6eed50694fd1b920b5b0d46891c9])
I20260924 19:57:52.949816 19297 tablet_loader.cc:96] loaded metadata for tablet a00e3626ea154dc5a959f0fbb4e849dc (table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3])
I20260924 19:57:52.969640 19297 tablet_loader.cc:96] loaded metadata for tablet a9e7c5b28fea4dbda802702c8b85ec54 (table default.Table_5208198cb618443dbec0f760012e4edd [id=7cb855f3577d4824ac7edb72b8f7072c])
W20260924 19:57:52.975025 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 54718.8us
I20260924 19:57:52.971004 19297 tablet_loader.cc:96] loaded metadata for tablet accddc7f696142bfa6a5e655f5abc220 (table default.Table_bdae18e3011a421d96cdcc107e8f1dc2 [id=ca7affed79d94a2597fdaa1ff38e5e32])
I20260924 19:57:52.924170 18224 raft_consensus.cc:359] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.926769 19474 raft_consensus.cc:515] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:52.977442 18224 raft_consensus.cc:385] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:52.977756 18224 raft_consensus.cc:740] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:52.978434 18224 consensus_queue.cc:260] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:52.979240 19474 leader_election.cc:290] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:52.982694 17521 leader_election.cc:336] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.527s (SENT)
I20260924 19:57:52.983170 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:52.991375 19297 tablet_loader.cc:96] loaded metadata for tablet affce4de8e0b412aa3b4bf19f9be7b6e (table default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06])
W20260924 19:57:52.994439 19524 log.cc:927] Time spent T 335f979eb04c49bcb53023693f54ad41 P f74e6ed351284d7fb110e44336ce643a: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20260924 19:57:52.998023 19297 tablet_loader.cc:96] loaded metadata for tablet b25dd98712d54cd5b53df00b2cf9f3c4 (table default.Table_e91b551877474bb0914b854a58447b3a [id=0606053a508145868b640f364fbf48d6])
I20260924 19:57:53.001709 19297 tablet_loader.cc:96] loaded metadata for tablet b474cde94649478b96809c7ef632f190 (table default.Table_b8cbd748636249c0ab17657b8f259893 [id=b9bee2849a334d098dd5d52fed7f6cc0])
I20260924 19:57:53.007112 19297 tablet_loader.cc:96] loaded metadata for tablet b7034626ce174d17bc82c92bfa6aebb3 (table default.Table_0a22025767fb456485c2624d5ca0dcbb [id=9f151548ce074cb6b8b8497fa246d61a])
I20260924 19:57:53.013785 19297 tablet_loader.cc:96] loaded metadata for tablet b7a1887d396f40a1a0c153dd0b2fe307 (table default.Table_954dba173a70455b90daee9230b44b6f [id=b743c3e3916a45069fabeed784e9e249])
I20260924 19:57:53.016626 18220 tablet_bootstrap.cc:492] T 2a1ac7e1b192465696640c02898bc39f P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:53.020020 19482 raft_consensus.cc:493] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:53.020560 19482 raft_consensus.cc:515] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:53.022823 18220 tablet_bootstrap.cc:654] T 2a1ac7e1b192465696640c02898bc39f P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:53.022841 19482 leader_election.cc:290] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:53.024838 17906 leader_election.cc:343] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 1d95d6e48173436c8c59130464413ea1
I20260924 19:57:53.027662 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ba365c977984e68a316356465b1e36a" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:53.028307 17970 raft_consensus.cc:2468] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:53.029241 19525 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 19:57:53.030575 17521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.188:38761, user_credentials={real_user=slave}} blocked reactor thread for 47917.7us
I20260924 19:57:53.031630 18050 leader_election.cc:304] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:53.032212 19297 tablet_loader.cc:96] loaded metadata for tablet bafaab10ce564b1b97659eccc0ab0967 (table default.Table_081e10d31dc74797abba37099d8a75fc [id=0ea22c5631c347aa9c7aa0e24140cb14])
W20260924 19:57:53.035573 17753 leader_election.cc:343] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:53.040203 17753 leader_election.cc:304] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:53.042436 19474 raft_consensus.cc:2749] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:53.053202 19297 tablet_loader.cc:96] loaded metadata for tablet bea1ebdf647e4ef9a131a0398672b8ab (table default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df])
I20260924 19:57:53.054522 19297 tablet_loader.cc:96] loaded metadata for tablet bf025253c49a4c23b79a25333cad160e (table default.Table_1d9dfc2ce03046ba955d3d82049402fd [id=c44dfe41598f4e6cbf320e3044b49fe5])
I20260924 19:57:53.059412 18224 ts_tablet_manager.cc:1434] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.150s user 0.008s sys 0.000s
I20260924 19:57:53.072068 18224 tablet_bootstrap.cc:492] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:53.077135 18220 tablet_bootstrap.cc:492] T 2a1ac7e1b192465696640c02898bc39f P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:53.077592 18220 ts_tablet_manager.cc:1403] T 2a1ac7e1b192465696640c02898bc39f P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.001s
I20260924 19:57:53.088128 19297 tablet_loader.cc:96] loaded metadata for tablet bf2650dbcae54dcba04f0078ae6c3e1d (table default.Table_5128840101ca460b9e13dfd054764e9d [id=e8a1bc657d9b4fb2b1f0eb458d41dbbd])
I20260924 19:57:53.091037 19474 raft_consensus.cc:493] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:53.091718 19474 raft_consensus.cc:515] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.099759 19474 leader_election.cc:290] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:53.115825 17753 leader_election.cc:343] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:53.116434 17753 leader_election.cc:304] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:53.118527 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ba365c977984e68a316356465b1e36a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:53.119381 17969 raft_consensus.cc:2468] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:53.108974 18220 raft_consensus.cc:359] T 2a1ac7e1b192465696640c02898bc39f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:53.120857 18220 raft_consensus.cc:385] T 2a1ac7e1b192465696640c02898bc39f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:53.121165 18220 raft_consensus.cc:740] T 2a1ac7e1b192465696640c02898bc39f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:53.121052 19474 raft_consensus.cc:2749] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:53.121878 18220 consensus_queue.cc:260] T 2a1ac7e1b192465696640c02898bc39f P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:53.091118 19297 tablet_loader.cc:96] loaded metadata for tablet c0feecd2fdb4409f86f6a5fbfbab3537 (table default.Table_5e50f3dbbb674d658a5afac0ebcedaa6 [id=545488f05d324b748e42dae4a665f0d6])
I20260924 19:57:53.126868 19297 tablet_loader.cc:96] loaded metadata for tablet c1684aaf9d984649bff8c4b0b12044b9 (table default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe])
I20260924 19:57:53.139142 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ba365c977984e68a316356465b1e36a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:53.140695 18113 raft_consensus.cc:2468] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:53.147898 18220 ts_tablet_manager.cc:1434] T 2a1ac7e1b192465696640c02898bc39f P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.070s user 0.005s sys 0.003s
I20260924 19:57:53.150815 19297 tablet_loader.cc:96] loaded metadata for tablet ca37ddd22f5d4bcab2862553408447ce (table default.Table_ca51e47e4fc44d9ba5dd07cbeefa6e68 [id=6ea0f9084fd84e848fdecdd696cdc773])
I20260924 19:57:53.151463 19403 raft_consensus.cc:2804] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:53.154132 19403 raft_consensus.cc:493] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:53.154631 19403 raft_consensus.cc:3060] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:53.149039 17969 raft_consensus.cc:3060] T cfe0303596204e528585c4fe1c476d8b P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:53.155529 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a1ac7e1b192465696640c02898bc39f" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:53.157801 17970 raft_consensus.cc:3060] T 2a1ac7e1b192465696640c02898bc39f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:53.158008 18220 tablet_bootstrap.cc:492] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:53.170744 19297 tablet_loader.cc:96] loaded metadata for tablet cba21f2e72914bc28a582433381a83f5 (table default.Table_9259880c400d44d69831fe68d31b06f4 [id=4f8c78c1109f4245b3ff41258543de1e])
I20260924 19:57:53.173626 17970 raft_consensus.cc:2468] T 2a1ac7e1b192465696640c02898bc39f P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:53.181046 18223 tablet_bootstrap.cc:492] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:53.181756 18223 ts_tablet_manager.cc:1403] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.461s user 0.004s sys 0.017s
I20260924 19:57:53.196261 18223 raft_consensus.cc:359] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:53.196722 19297 tablet_loader.cc:96] loaded metadata for tablet cfe0303596204e528585c4fe1c476d8b (table default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265])
I20260924 19:57:53.197106 18223 raft_consensus.cc:385] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:53.197394 18223 raft_consensus.cc:740] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:53.198024 18223 consensus_queue.cc:260] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:53.211510 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a1ac7e1b192465696640c02898bc39f" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:53.212143 18113 raft_consensus.cc:3060] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:53.219746 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8523125e35554830aff04b17677d5ceb" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:53.224236 19297 tablet_loader.cc:96] loaded metadata for tablet d129e0979a584f40ba9a975349cf15a5 (table default.Table_a12f8684de2041ec83f2fee77829afdd [id=94cea8de30ca4010a95f142ac8408653])
I20260924 19:57:53.224629 18224 tablet_bootstrap.cc:654] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:53.234745 18220 tablet_bootstrap.cc:654] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:53.240947 19297 tablet_loader.cc:96] loaded metadata for tablet d81f74c7ba2443e49228b17e867588cc (table default.Table_e3a6abe9c34a4fd6a8567a9c0a8190c3 [id=4e80d69486a34b848393f01e656d122b])
I20260924 19:57:53.248329 19297 tablet_loader.cc:96] loaded metadata for tablet d930e7f4a6b7471899dab1b28e40f93d (table default.Table_9b8f6ad547fc4c3fa379c93590aebd56 [id=4fc8e565a6a640549716cee596fa3f9d])
I20260924 19:57:53.260538 18223 ts_tablet_manager.cc:1434] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.078s user 0.005s sys 0.002s
I20260924 19:57:53.261730 18223 tablet_bootstrap.cc:492] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:53.265558 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 52755.8us
I20260924 19:57:53.271704 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ba365c977984e68a316356465b1e36a" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:53.272449 17816 raft_consensus.cc:2468] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:53.285912 19297 tablet_loader.cc:96] loaded metadata for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140])
I20260924 19:57:53.289058 19297 tablet_loader.cc:96] loaded metadata for tablet dd434e38c9364f55bd18383149015028 (table default.Table_a7d63063e7ee4493abe8c76da6d4f99d [id=be19c62729a34ae1a15966662303a869])
I20260924 19:57:53.290354 19297 tablet_loader.cc:96] loaded metadata for tablet df722b0bbfd04ce2943f0e5b8058c50f (table default.Table_db4a5e39fd954bb0b4c271899feefe90 [id=0b7e96c71a074f838a519746d9c92c3f])
I20260924 19:57:53.298593 19398 raft_consensus.cc:493] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:53.299981 19398 raft_consensus.cc:515] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.302197 19398 leader_election.cc:290] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:53.304594 18220 tablet_bootstrap.cc:492] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:53.306567 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accddc7f696142bfa6a5e655f5abc220" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:53.307228 17816 raft_consensus.cc:2468] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:53.308715 17906 leader_election.cc:304] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:53.309777 19398 raft_consensus.cc:2804] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:53.310171 19398 raft_consensus.cc:493] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:53.310619 19398 raft_consensus.cc:3060] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:53.319854 18220 ts_tablet_manager.cc:1403] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.162s user 0.010s sys 0.007s
I20260924 19:57:53.346439 19297 tablet_loader.cc:96] loaded metadata for tablet e0246d83189d4ba5b75fbd8390f517f6 (table default.Table_202c25f88e1a485c9ca6677d9cf53e75 [id=221fc6b9725747ada3ed73dc569052cb])
I20260924 19:57:53.349989 19297 tablet_loader.cc:96] loaded metadata for tablet e1137318a9fa4a0781f718dc34c4ecbe (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8])
I20260924 19:57:53.364184 18220 raft_consensus.cc:359] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:53.364833 18220 raft_consensus.cc:385] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:53.365087 18220 raft_consensus.cc:740] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:53.365797 18220 consensus_queue.cc:260] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:53.370540 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8523125e35554830aff04b17677d5ceb" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:53.382086 19530 consensus_queue.cc:1048] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:53.386912 18220 ts_tablet_manager.cc:1434] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.067s user 0.006s sys 0.000s
I20260924 19:57:53.388518 18220 tablet_bootstrap.cc:492] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:53.397078 19474 raft_consensus.cc:493] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:53.397558 19474 raft_consensus.cc:515] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.400085 19474 leader_election.cc:290] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:53.398267 19297 tablet_loader.cc:96] loaded metadata for tablet e59e737a68dd443f9898113b2b2a401e (table default.Table_1dad27c8fd4445fb89e32722eef0abe2 [id=4cd61b434f884eed908abf700d02cdd0])
I20260924 19:57:53.402498 19297 tablet_loader.cc:96] loaded metadata for tablet e68378312c334fc28dafb71bf6386be8 (table default.Table_8be3db507b2d42da90969bacc361c127 [id=cd5ff543cb1f48f58d5867aef9706963])
I20260924 19:57:53.402763 19403 raft_consensus.cc:515] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.403681 19297 tablet_loader.cc:96] loaded metadata for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_88352246addd43f4be871c983413b2de [id=fa469a44fb3741d39036337129cc36a1])
I20260924 19:57:53.405200 19403 leader_election.cc:290] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:53.406129 19297 tablet_loader.cc:96] loaded metadata for tablet f10625ef84054ca0babbc583395dd96c (table default.Table_8ade2ea47f344fd18a687f7d5e0ea568 [id=eecec62f4e6a4e56920d841de1b9077b])
W20260924 19:57:53.407181 18899 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.189:43689, user_credentials={real_user=slave}} blocked reactor thread for 82678.8us
I20260924 19:57:53.394933 18113 raft_consensus.cc:2468] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:53.410763 19398 raft_consensus.cc:515] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.420768 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accddc7f696142bfa6a5e655f5abc220" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:53.421496 17817 raft_consensus.cc:3060] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:53.424425 19297 tablet_loader.cc:96] loaded metadata for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803])
I20260924 19:57:53.426554 17753 leader_election.cc:304] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:53.427356 19474 raft_consensus.cc:2804] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:53.427762 19474 raft_consensus.cc:493] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:53.428145 19474 raft_consensus.cc:3060] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:53.428367 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accddc7f696142bfa6a5e655f5abc220" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:53.435520 19474 raft_consensus.cc:515] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.437638 19474 leader_election.cc:290] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:53.439967 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accddc7f696142bfa6a5e655f5abc220" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:53.442157 19297 tablet_loader.cc:96] loaded metadata for tablet f43f2a255c644ca6a68a8d867798fc1a (table default.Table_4e3a3158cde24302806b7ba7827831c4 [id=fa0b10df171340868b0305a28d192ea5])
I20260924 19:57:53.448920 19297 tablet_loader.cc:96] loaded metadata for tablet fbd5915ee98a4eda94479e9330cf8d04 (table default.Table_83714385573a44b2af65ac658fbe2cfa [id=e5b211f0af7143709eec620ecf04f2b5])
I20260924 19:57:53.450147 19297 tablet_loader.cc:96] loaded metadata for tablet fd6e656a653a43d1843a91fee813237d (table default.Table_e3179b11ea114ce99779ed0b2ac46732 [id=8c39c8f34038451d840db3c909d76f3d])
W20260924 19:57:53.455597 17905 leader_election.cc:343] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 1d95d6e48173436c8c59130464413ea1
I20260924 19:57:53.455991 17905 leader_election.cc:304] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:53.456624 19336 raft_consensus.cc:2749] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:53.457201 18220 tablet_bootstrap.cc:654] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:53.460016 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ba365c977984e68a316356465b1e36a" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:53.460731 17969 raft_consensus.cc:3060] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:53.476800 19398 leader_election.cc:290] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:53.481534 17543 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:53.454005 18224 tablet_bootstrap.cc:492] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:53.483315 18224 ts_tablet_manager.cc:1403] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.411s user 0.006s sys 0.015s
I20260924 19:57:53.483700 17542 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
W20260924 19:57:53.484599 17542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 406) took 3754 ms (client timeout 582 ms). Trace:
W20260924 19:57:53.485015 17542 rpcz_store.cc:269] 0924 19:57:49.730483 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.730714 (+ 231us) service_pool.cc:224] Handling call
0924 19:57:53.484493 (+3753779us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260924 19:57:53.481419 19297 catalog_manager.cc:1525] Time spent T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6: Loading table and tablet metadata into memory: real 5.246s user 0.365s sys 0.057s
I20260924 19:57:53.486630 19297 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260924 19:57:53.485853 18224 raft_consensus.cc:359] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:53.451685 17753 leader_election.cc:343] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:53.466485 19472 raft_consensus.cc:493] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:53.487331 17540 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:53.487396 19472 raft_consensus.cc:515] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:53.488217 17540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 407) took 2381 ms (client timeout 574 ms). Trace:
W20260924 19:57:53.488586 17540 rpcz_store.cc:269] 0924 19:57:51.107343 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:51.107534 (+ 191us) service_pool.cc:224] Handling call
0924 19:57:53.488161 (+2380627us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:53.489216 19472 leader_election.cc:290] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:53.489552 17540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58098 (request call id 7) took 801 ms (client timeout 636 ms). Trace:
I20260924 19:57:53.490411 17753 leader_election.cc:304] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:53.491122 19472 raft_consensus.cc:2804] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:53.491629 19472 raft_consensus.cc:697] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:53.492489 19472 consensus_queue.cc:237] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:53.493038 17541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
W20260924 19:57:53.494009 17541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58098 (request call id 5) took 2718 ms (client timeout 528 ms). Trace:
W20260924 19:57:53.494390 17541 rpcz_store.cc:269] 0924 19:57:50.775716 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:50.775956 (+ 240us) service_pool.cc:224] Handling call
0924 19:57:53.493949 (+2717993us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:53.495301 17541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:53.496464 17545 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
W20260924 19:57:53.497316 17545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58098 (request call id 3) took 4407 ms (client timeout 500 ms). Trace:
W20260924 19:57:53.497666 17545 rpcz_store.cc:269] 0924 19:57:49.090239 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.091255 (+ 1016us) service_pool.cc:224] Handling call
0924 19:57:53.497239 (+4405984us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:53.498581 17537 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
W20260924 19:57:53.499467 17537 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 409) took 1811 ms (client timeout 574 ms). Trace:
W20260924 19:57:53.499850 17537 rpcz_store.cc:269] 0924 19:57:51.688356 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:51.688569 (+ 213us) service_pool.cc:224] Handling call
0924 19:57:53.499380 (+1810811us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:53.500872 17539 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6"
W20260924 19:57:53.501776 17539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 408) took 2197 ms (client timeout 707 ms). Trace:
W20260924 19:57:53.502213 17539 rpcz_store.cc:269] 0924 19:57:51.304163 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:51.304405 (+ 242us) service_pool.cc:224] Handling call
0924 19:57:53.501717 (+2197312us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:53.503167 17538 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:53.504225 18223 tablet_bootstrap.cc:654] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:53.504169 17538 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58098 (request call id 6) took 2147 ms (client timeout 528 ms). Trace:
W20260924 19:57:53.504797 17601 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet df722b0bbfd04ce2943f0e5b8058c50f (table default.Table_db4a5e39fd954bb0b4c271899feefe90 [id=0b7e96c71a074f838a519746d9c92c3f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet df722b0bbfd04ce2943f0e5b8058c50f consensus state has no leader
W20260924 19:57:53.504604 17538 rpcz_store.cc:269] 0924 19:57:51.356331 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:51.356578 (+ 247us) service_pool.cc:224] Handling call
0924 19:57:53.504104 (+2147526us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:53.483026 17544 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" candidate_term: 5 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
W20260924 19:57:53.507372 17544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 405) took 4442 ms (client timeout 582 ms). Trace:
W20260924 19:57:53.507721 17544 rpcz_store.cc:269] 0924 19:57:49.065107 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.065239 (+ 132us) service_pool.cc:224] Handling call
0924 19:57:53.507308 (+4442069us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:57:53.487087 18224 raft_consensus.cc:385] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:53.508961 18224 raft_consensus.cc:740] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:53.509472 17817 raft_consensus.cc:2468] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:53.510756 17906 leader_election.cc:304] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:53.512421 19537 raft_consensus.cc:2804] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:53.513191 19537 raft_consensus.cc:697] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:53.482400 17543 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e059bbe4dac4178bccb3be48cb1118d in term 5.
W20260924 19:57:53.515131 17543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58098 (request call id 4) took 3884 ms (client timeout 500 ms). Trace:
W20260924 19:57:53.515558 17543 rpcz_store.cc:269] 0924 19:57:49.630563 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.630761 (+ 198us) service_pool.cc:224] Handling call
0924 19:57:53.515052 (+3884291us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260924 19:57:53.515916 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 58566.3us
I20260924 19:57:53.517607 17536 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Refusing update from remote peer e1ff67c5d57d474a9d92cc4c6de1287e: Log matching property violated. Preceding OpId in replica: term: 4 index: 322. Preceding OpId from leader: term: 5 index: 323. (index mismatch)
W20260924 19:57:53.518425 17905 leader_election.cc:343] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:53.519369 17905 leader_election.cc:343] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:53.533213 17534 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:36514 (request call id 1) took 4113 ms. Trace:
I20260924 19:57:53.533558 17534 rpcz_store.cc:276] 0924 19:57:49.419577 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:49.421193 (+ 1616us) service_pool.cc:224] Handling call
0924 19:57:53.532791 (+4111598us) spinlock_profiling.cc:243] Waited 48.1 ms on lock 0x7b5400030554. stack: 00007fce73caa323 00007fce73caa220 00007fce7384e574 0000000000516964 00000000005168f9 00007fce7ada7aa1 00007fce7ae477b9 00007fce7cc75c76 00007fce7ccb9027 00007fce7a85e534 00007fce7a85e4c2 00007fce7a85e3f1 00007fce7a85e36c 00007fce7a85d622 00007fce78b0d09c 00007fce78b0c4d6
0924 19:57:53.533102 (+ 311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":110545152}
W20260924 19:57:53.492403 17540 rpcz_store.cc:269] 0924 19:57:52.687717 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:53.489337 (+801620us) service_pool.cc:211] Skipping call since client already timed out
0924 19:57:53.489435 (+ 98us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260924 19:57:53.537262 17536 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42048 (request call id 410) took 1706 ms. Trace:
I20260924 19:57:53.537624 17536 rpcz_store.cc:276] 0924 19:57:51.831101 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:51.831328 (+ 227us) service_pool.cc:224] Handling call
0924 19:57:53.516550 (+1685222us) raft_consensus.cc:1471] Updating replica for 0 ops
0924 19:57:53.517934 (+ 1384us) raft_consensus.cc:1697] Filling consensus response to leader.
0924 19:57:53.519939 (+ 2005us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b5400030550. stack: 00007fce73caa323 00007fce73caa220 00007fce7384e574 0000000000516964 00000000005168f9 00007fce7ada7aa1 00007fce7ae3e06d 00007fce7be9e659 00007fce7a51ad34 00007fce7a51acc2 00007fce7a51abf1 00007fce7a51ab6c 00007fce7a519e22 00007fce78b0d09c 00007fce78b0c4d6 00007fce78b0be7f
0924 19:57:53.537194 (+ 17255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24823680}
W20260924 19:57:53.486425 17542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42048 (request call id 411) took 1473 ms (client timeout 707 ms). Trace:
I20260924 19:57:53.543942 19489 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.001s
W20260924 19:57:53.543367 17542 rpcz_store.cc:269] 0924 19:57:52.012791 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:53.486144 (+1473353us) service_pool.cc:211] Skipping call since client already timed out
0924 19:57:53.486359 (+ 215us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260924 19:57:53.514179 19537 consensus_queue.cc:237] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.556200 19297 catalog_manager.cc:1308] Loaded cluster ID: 08019495472c40618d8cccdc88d885cc
I20260924 19:57:53.556545 19297 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260924 19:57:53.567065 18224 consensus_queue.cc:260] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:53.579327 18053 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.17.2.190:46309, user_credentials={real_user=slave}} blocked reactor thread for 48711.4us
I20260924 19:57:53.589159 18223 tablet_bootstrap.cc:492] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:53.589681 18223 ts_tablet_manager.cc:1403] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.328s user 0.014s sys 0.005s
I20260924 19:57:53.590556 18220 tablet_bootstrap.cc:492] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:53.590981 18220 ts_tablet_manager.cc:1403] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.203s user 0.002s sys 0.007s
I20260924 19:57:53.612998 18224 ts_tablet_manager.cc:1434] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.129s user 0.001s sys 0.006s
I20260924 19:57:53.613888 19297 catalog_manager.cc:1553] Loading token signing keys...
W20260924 19:57:53.613935 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 108515us
W20260924 19:57:53.614681 17753 leader_election.cc:343] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:53.615137 17753 leader_election.cc:304] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:53.599145 17969 raft_consensus.cc:2468] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:53.617394 19472 raft_consensus.cc:2749] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:53.619306 19472 consensus_queue.cc:1048] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:53.611523 18223 raft_consensus.cc:359] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.624866 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd5915ee98a4eda94479e9330cf8d04" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:53.623754 18223 raft_consensus.cc:385] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:53.631474 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ba365c977984e68a316356465b1e36a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:53.632349 17969 raft_consensus.cc:2393] T 7ba365c977984e68a316356465b1e36a P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c4f6e3a7b06148918d230c77ab0c8202 in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:53.627364 18223 raft_consensus.cc:740] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:53.644487 18220 raft_consensus.cc:359] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.645205 18220 raft_consensus.cc:385] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:53.645480 18220 raft_consensus.cc:740] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:53.644074 18223 consensus_queue.cc:260] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.664788 18224 tablet_bootstrap.cc:492] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:53.661490 19541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ff67c5d57d474a9d92cc4c6de1287e. Latest consensus state: current_term: 5 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:53.672288 19541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:53.675843 19546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:53.679345 19546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:57:53.689440 18224 tablet_bootstrap.cc:654] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:53.710382 18223 ts_tablet_manager.cc:1434] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.120s user 0.010s sys 0.001s
I20260924 19:57:53.711499 18223 tablet_bootstrap.cc:492] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:53.712294 19497 raft_consensus.cc:493] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:53.712741 19497 raft_consensus.cc:515] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.680480 18050 leader_election.cc:304] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:53.715446 19297 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6: Loaded TSK: 0
I20260924 19:57:53.716346 19408 raft_consensus.cc:2804] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:53.716897 19408 raft_consensus.cc:697] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:53.717876 19497 leader_election.cc:290] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:53.717820 19408 consensus_queue.cc:237] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.723774 19297 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260924 19:57:53.701112 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c058a49bfd5544b58b76edfd3fadbd1a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:53.651381 18220 consensus_queue.cc:260] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.727066 17969 raft_consensus.cc:2468] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:53.730660 18220 ts_tablet_manager.cc:1434] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.139s user 0.002s sys 0.003s
I20260924 19:57:53.731760 18220 tablet_bootstrap.cc:492] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:53.745530 18223 tablet_bootstrap.cc:654] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:53.760136 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ba365c977984e68a316356465b1e36a" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:53.760875 17816 raft_consensus.cc:2393] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eaf64bfb64aa42948c9bc1411ce4ce35 in current term 1: Already voted for candidate c4f6e3a7b06148918d230c77ab0c8202 in this term.
W20260924 19:57:53.783603 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 59697us
W20260924 19:57:53.811939 19297 catalog_manager.cc:1458] Failed waiting for node to catch up after master election: Illegal state: Node 6f88e00b235f48bb82e6a3c9c9f51fa6 not leader. Raft Consensus state: current_term: 5 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
W20260924 19:57:53.812737 19297 catalog_manager.cc:1458] Failed waiting for node to catch up after master election: Illegal state: Node 6f88e00b235f48bb82e6a3c9c9f51fa6 not leader. Raft Consensus state: current_term: 5 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:53.822374 18224 tablet_bootstrap.cc:492] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:53.822834 18224 ts_tablet_manager.cc:1403] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.158s user 0.014s sys 0.013s
I20260924 19:57:53.823956 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd5915ee98a4eda94479e9330cf8d04" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:53.826485 18224 raft_consensus.cc:359] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.827097 18224 raft_consensus.cc:385] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:53.827361 18224 raft_consensus.cc:740] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:53.827917 18224 consensus_queue.cc:260] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.829854 18224 ts_tablet_manager.cc:1434] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260924 19:57:53.831791 18224 tablet_bootstrap.cc:492] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:53.835563 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7ba365c977984e68a316356465b1e36a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:53.837504 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c058a49bfd5544b58b76edfd3fadbd1a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:53.840965 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accddc7f696142bfa6a5e655f5abc220" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:53.844017 18223 tablet_bootstrap.cc:492] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:53.844422 18223 ts_tablet_manager.cc:1403] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.133s user 0.004s sys 0.009s
I20260924 19:57:53.848094 18223 raft_consensus.cc:359] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.848726 18223 raft_consensus.cc:385] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:53.848985 18223 raft_consensus.cc:740] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:53.849615 18223 consensus_queue.cc:260] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.850555 18224 tablet_bootstrap.cc:654] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:53.851645 18223 ts_tablet_manager.cc:1434] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 19:57:53.852524 18223 tablet_bootstrap.cc:492] T c058a49bfd5544b58b76edfd3fadbd1a P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:53.860647 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "accddc7f696142bfa6a5e655f5abc220" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:53.861388 17817 raft_consensus.cc:2393] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eaf64bfb64aa42948c9bc1411ce4ce35 in current term 1: Already voted for candidate f74e6ed351284d7fb110e44336ce643a in this term.
W20260924 19:57:53.881346 17753 leader_election.cc:343] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:53.882236 17753 leader_election.cc:343] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:53.882937 17753 leader_election.cc:343] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:53.883275 17753 leader_election.cc:304] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:53.883878 19472 raft_consensus.cc:2749] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:53.884200 17753 leader_election.cc:304] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:53.885051 19472 raft_consensus.cc:2749] T 7ba365c977984e68a316356465b1e36a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 19:57:53.885347 17753 leader_election.cc:304] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters: eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:53.885985 19472 raft_consensus.cc:2804] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:53.886322 19472 raft_consensus.cc:493] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:53.886727 19472 raft_consensus.cc:3060] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:53.896646 18224 tablet_bootstrap.cc:492] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:53.897179 18224 ts_tablet_manager.cc:1403] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.003s
I20260924 19:57:53.900445 18224 raft_consensus.cc:359] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:53.902724 18224 raft_consensus.cc:385] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:53.903056 18224 raft_consensus.cc:740] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:53.903815 18224 consensus_queue.cc:260] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:53.913144 18220 tablet_bootstrap.cc:654] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:53.925428 19472 raft_consensus.cc:515] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.936169 18050 leader_election.cc:304] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:53.937682 19500 raft_consensus.cc:3060] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:53.944140 19555 raft_consensus.cc:493] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:53.944782 19500 raft_consensus.cc:2749] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:53.944672 19555 raft_consensus.cc:515] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:53.947485 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57102b1727e94a90981509c82c082441" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:53.961715 17753 leader_election.cc:343] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:53.963557 18223 tablet_bootstrap.cc:654] T c058a49bfd5544b58b76edfd3fadbd1a P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:53.974745 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c058a49bfd5544b58b76edfd3fadbd1a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:53.975347 17969 raft_consensus.cc:3060] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:53.992522 18224 ts_tablet_manager.cc:1434] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.095s user 0.017s sys 0.001s
I20260924 19:57:53.998860 18223 tablet_bootstrap.cc:492] T c058a49bfd5544b58b76edfd3fadbd1a P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:53.999334 18223 ts_tablet_manager.cc:1403] T c058a49bfd5544b58b76edfd3fadbd1a P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.147s user 0.010s sys 0.000s
I20260924 19:57:53.997748 19555 leader_election.cc:290] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:54.001854 18223 raft_consensus.cc:359] T c058a49bfd5544b58b76edfd3fadbd1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.002480 18223 raft_consensus.cc:385] T c058a49bfd5544b58b76edfd3fadbd1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.002727 18223 raft_consensus.cc:740] T c058a49bfd5544b58b76edfd3fadbd1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.004742 17969 raft_consensus.cc:2468] T c058a49bfd5544b58b76edfd3fadbd1a P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:54.015455 18223 consensus_queue.cc:260] T c058a49bfd5544b58b76edfd3fadbd1a P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.021564 18220 tablet_bootstrap.cc:492] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:54.022073 18220 ts_tablet_manager.cc:1403] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.291s user 0.006s sys 0.005s
I20260924 19:57:54.024524 18220 raft_consensus.cc:359] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.035769 18220 raft_consensus.cc:385] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.036114 18220 raft_consensus.cc:740] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.038930 18223 ts_tablet_manager.cc:1434] T c058a49bfd5544b58b76edfd3fadbd1a P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.039s user 0.004s sys 0.001s
I20260924 19:57:54.039945 18223 tablet_bootstrap.cc:492] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:54.045506 18903 leader_election.cc:336] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.636s (SENT)
I20260924 19:57:54.045969 18903 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:54.046235 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c058a49bfd5544b58b76edfd3fadbd1a" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:54.006137 17753 leader_election.cc:304] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:53.997503 19472 leader_election.cc:290] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:54.051450 18110 raft_consensus.cc:3060] T c058a49bfd5544b58b76edfd3fadbd1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:57:54.051709 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 45659.3us
I20260924 19:57:54.051801 19472 raft_consensus.cc:2804] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:54.052446 19472 raft_consensus.cc:697] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:54.056403 18224 tablet_bootstrap.cc:492] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:54.059108 19472 consensus_queue.cc:237] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.061160 18110 raft_consensus.cc:2468] T c058a49bfd5544b58b76edfd3fadbd1a P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:54.061982 18224 tablet_bootstrap.cc:654] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.077632 18223 tablet_bootstrap.cc:654] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.085693 18224 tablet_bootstrap.cc:492] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:54.086133 18224 ts_tablet_manager.cc:1403] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.003s
I20260924 19:57:54.095553 18224 raft_consensus.cc:359] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.096215 18224 raft_consensus.cc:385] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.096464 18224 raft_consensus.cc:740] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.079983 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57102b1727e94a90981509c82c082441" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:54.100848 18224 consensus_queue.cc:260] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.109722 19562 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } } }
I20260924 19:57:54.110482 18224 ts_tablet_manager.cc:1434] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20260924 19:57:54.110551 19562 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e [sys.catalog]: This master's current role is: LEADER
I20260924 19:57:54.082472 19561 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 19:57:54.113759 18224 tablet_bootstrap.cc:492] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:54.036957 18220 consensus_queue.cc:260] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.138559 18224 tablet_bootstrap.cc:654] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.139317 19472 consensus_queue.cc:1048] T b25dd98712d54cd5b53df00b2cf9f3c4 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:54.142819 18220 ts_tablet_manager.cc:1434] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.120s user 0.003s sys 0.002s
I20260924 19:57:54.167999 18220 tablet_bootstrap.cc:492] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:54.171805 18223 tablet_bootstrap.cc:492] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:54.172226 18223 ts_tablet_manager.cc:1403] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.132s user 0.011s sys 0.006s
I20260924 19:57:54.173907 19500 raft_consensus.cc:493] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.174525 19500 raft_consensus.cc:515] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:54.176360 18224 tablet_bootstrap.cc:492] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:54.176789 18224 ts_tablet_manager.cc:1403] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.004s
I20260924 19:57:54.177232 19500 leader_election.cc:290] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:54.175788 18223 raft_consensus.cc:359] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.177901 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c1afbfc7de7480ab4be40680a3e3ada" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:54.178236 18223 raft_consensus.cc:385] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.178506 17817 raft_consensus.cc:2468] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:54.182797 18224 raft_consensus.cc:359] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.183490 18224 raft_consensus.cc:385] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.183761 18224 raft_consensus.cc:740] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.184587 19472 raft_consensus.cc:493] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.184513 18224 consensus_queue.cc:260] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.185031 19472 raft_consensus.cc:515] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.186959 18224 ts_tablet_manager.cc:1434] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.010s user 0.004s sys 0.003s
I20260924 19:57:54.187842 18224 tablet_bootstrap.cc:492] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:54.187659 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2de3877b94b44c94b6d0fdad93f8aa79" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:54.188969 17753 leader_election.cc:343] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.188685 19472 leader_election.cc:290] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:54.179785 18051 leader_election.cc:304] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:54.195060 19403 raft_consensus.cc:2804] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:54.195451 19403 raft_consensus.cc:493] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:54.195763 19403 raft_consensus.cc:3060] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.204566 18224 tablet_bootstrap.cc:654] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.205228 19403 raft_consensus.cc:515] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:54.207242 18223 raft_consensus.cc:740] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.205161 19472 raft_consensus.cc:493] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.208145 18223 consensus_queue.cc:260] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.215518 19403 leader_election.cc:290] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:54.216743 18223 ts_tablet_manager.cc:1434] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.044s user 0.006s sys 0.000s
I20260924 19:57:54.225181 18223 tablet_bootstrap.cc:492] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:54.208242 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c1afbfc7de7480ab4be40680a3e3ada" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:54.226706 17817 raft_consensus.cc:3060] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.232256 18223 tablet_bootstrap.cc:654] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.237710 18224 tablet_bootstrap.cc:492] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:54.238219 18224 ts_tablet_manager.cc:1403] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.001s
I20260924 19:57:54.215620 19472 raft_consensus.cc:515] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.240650 18224 raft_consensus.cc:359] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.241259 18224 raft_consensus.cc:385] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.241496 18224 raft_consensus.cc:740] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.242219 19472 leader_election.cc:290] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:54.242079 18224 consensus_queue.cc:260] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.242806 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c05ec2ba01448d998c6ffa1525de984" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:54.243430 17969 raft_consensus.cc:2468] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:54.244916 17753 leader_election.cc:304] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:54.245930 19557 raft_consensus.cc:2804] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:54.246234 19557 raft_consensus.cc:493] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:54.246537 19557 raft_consensus.cc:3060] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.252934 19557 raft_consensus.cc:515] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.256757 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c05ec2ba01448d998c6ffa1525de984" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:54.257329 17969 raft_consensus.cc:3060] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.259240 19557 leader_election.cc:290] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:54.259298 19542 raft_consensus.cc:493] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.262841 18223 tablet_bootstrap.cc:492] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:54.263253 18223 ts_tablet_manager.cc:1403] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.015s
I20260924 19:57:54.265648 18223 raft_consensus.cc:359] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.266242 18223 raft_consensus.cc:385] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.266541 18223 raft_consensus.cc:740] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.266314 17969 raft_consensus.cc:2468] T 8c05ec2ba01448d998c6ffa1525de984 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:54.259840 19542 raft_consensus.cc:515] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:54.267138 18223 consensus_queue.cc:260] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.268682 17753 leader_election.cc:304] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:54.269080 19542 leader_election.cc:290] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:54.271431 18224 ts_tablet_manager.cc:1434] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.033s user 0.007s sys 0.001s
I20260924 19:57:54.271946 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c1afbfc7de7480ab4be40680a3e3ada" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:54.272502 17966 raft_consensus.cc:2468] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:54.274124 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c1afbfc7de7480ab4be40680a3e3ada" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:54.274591 17966 raft_consensus.cc:3060] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.277149 18223 ts_tablet_manager.cc:1434] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.014s user 0.000s sys 0.004s
I20260924 19:57:54.277997 18223 tablet_bootstrap.cc:492] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:54.281229 17966 raft_consensus.cc:2468] T 7c1afbfc7de7480ab4be40680a3e3ada P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:54.281272 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "144355b18c2f413baa28dd632623db52" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:54.272742 17753 leader_election.cc:343] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.283036 17753 leader_election.cc:304] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:54.284162 17753 leader_election.cc:343] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.284233 18224 tablet_bootstrap.cc:492] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:54.286595 19557 raft_consensus.cc:2804] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:54.288349 19557 raft_consensus.cc:697] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 LEADER]: Becoming Leader. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Running, Role: LEADER
I20260924 19:57:54.289120 19557 consensus_queue.cc:237] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.291337 19555 raft_consensus.cc:2749] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:54.294458 18224 tablet_bootstrap.cc:654] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.320156 18224 tablet_bootstrap.cc:492] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:54.320566 18224 ts_tablet_manager.cc:1403] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.002s
I20260924 19:57:54.327698 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2de3877b94b44c94b6d0fdad93f8aa79" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:54.331363 18224 raft_consensus.cc:359] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:54.332146 18224 raft_consensus.cc:385] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.332428 18224 raft_consensus.cc:740] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.335373 18224 consensus_queue.cc:260] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:54.343801 18224 ts_tablet_manager.cc:1434] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260924 19:57:54.344735 18224 tablet_bootstrap.cc:492] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:54.350493 18224 tablet_bootstrap.cc:654] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.355432 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c05ec2ba01448d998c6ffa1525de984" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:54.356000 18113 raft_consensus.cc:2468] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 0.
I20260924 19:57:54.361119 18220 tablet_bootstrap.cc:654] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.366394 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c05ec2ba01448d998c6ffa1525de984" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:54.366983 18107 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "144355b18c2f413baa28dd632623db52" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:54.367005 18113 raft_consensus.cc:3060] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.372044 18223 tablet_bootstrap.cc:654] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:54.380280 17753 leader_election.cc:343] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.381747 17753 leader_election.cc:304] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:54.383381 19557 raft_consensus.cc:2749] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:54.385586 18224 tablet_bootstrap.cc:492] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:54.386027 18224 ts_tablet_manager.cc:1403] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.000s
I20260924 19:57:54.388180 19557 raft_consensus.cc:493] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.388640 19557 raft_consensus.cc:515] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.390259 18050 leader_election.cc:304] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:54.390754 19557 leader_election.cc:290] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:54.391381 19375 raft_consensus.cc:2804] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:54.391872 18224 raft_consensus.cc:359] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.392590 18224 raft_consensus.cc:385] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.392864 18224 raft_consensus.cc:740] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.393599 18224 consensus_queue.cc:260] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.396941 19375 raft_consensus.cc:697] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:54.397956 19375 consensus_queue.cc:237] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:54.398617 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59e737a68dd443f9898113b2b2a401e" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:54.399112 17753 leader_election.cc:343] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.399628 17753 leader_election.cc:304] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:54.400761 18223 tablet_bootstrap.cc:492] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:54.401132 18223 ts_tablet_manager.cc:1403] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.123s user 0.005s sys 0.007s
W20260924 19:57:54.402127 17878 tablet.cc:2433] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:57:54.403827 18223 raft_consensus.cc:359] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.404472 18223 raft_consensus.cc:385] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.404732 18223 raft_consensus.cc:740] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.405346 18223 consensus_queue.cc:260] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.412564 18223 ts_tablet_manager.cc:1434] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260924 19:57:54.396351 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59e737a68dd443f9898113b2b2a401e" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:54.420215 19557 raft_consensus.cc:2749] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:54.422204 18224 ts_tablet_manager.cc:1434] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20260924 19:57:54.423202 18224 tablet_bootstrap.cc:492] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:54.426683 19557 consensus_queue.cc:1048] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:54.428225 18223 tablet_bootstrap.cc:492] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:54.428412 17753 leader_election.cc:343] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.440171 17817 raft_consensus.cc:2468] T 7c1afbfc7de7480ab4be40680a3e3ada P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:54.442819 19472 raft_consensus.cc:493] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.443356 19472 raft_consensus.cc:515] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.445464 19472 leader_election.cc:290] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:54.446175 17753 leader_election.cc:343] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.446583 17753 leader_election.cc:304] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:54.454763 18223 tablet_bootstrap.cc:654] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.454923 18224 tablet_bootstrap.cc:654] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.456182 19375 raft_consensus.cc:493] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.457034 19375 raft_consensus.cc:515] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.459409 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18bb8f3d43df4aae8614b6dba9ffc668" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:54.459769 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18bb8f3d43df4aae8614b6dba9ffc668" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:54.459751 19375 leader_election.cc:290] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:54.461750 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7635c52883ec4434bad0cc15f8733f72" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:54.463044 18050 leader_election.cc:343] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.455346 19472 raft_consensus.cc:2749] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:54.463318 19557 raft_consensus.cc:493] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.467780 19557 raft_consensus.cc:515] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.470023 19557 leader_election.cc:290] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:57:54.463960 17753 leader_election.cc:343] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:54.470860 17753 leader_election.cc:343] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.471315 17753 leader_election.cc:304] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:54.472019 19557 raft_consensus.cc:2749] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:54.475674 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97bbb17c170d42349d2c9cdb41349cb9" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:54.475900 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7635c52883ec4434bad0cc15f8733f72" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:54.476538 17818 raft_consensus.cc:2468] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:54.479178 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97bbb17c170d42349d2c9cdb41349cb9" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:54.485386 18170 tablet.cc:2433] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:57:54.487613 18051 leader_election.cc:304] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:54.489494 17753 leader_election.cc:343] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.489578 19529 raft_consensus.cc:2804] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:54.489882 19529 raft_consensus.cc:493] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:54.490206 19529 raft_consensus.cc:3060] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.501411 19529 raft_consensus.cc:515] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:54.502537 17753 leader_election.cc:343] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.506440 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7635c52883ec4434bad0cc15f8733f72" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:54.502956 17753 leader_election.cc:304] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
W20260924 19:57:54.508055 18050 leader_election.cc:343] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.507431 19529 leader_election.cc:290] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:54.511276 19557 raft_consensus.cc:2749] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:54.518164 18224 tablet_bootstrap.cc:492] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:54.518649 18224 ts_tablet_manager.cc:1403] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.096s user 0.011s sys 0.000s
I20260924 19:57:54.522138 18220 tablet_bootstrap.cc:492] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:54.522564 18220 ts_tablet_manager.cc:1403] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.355s user 0.006s sys 0.004s
I20260924 19:57:54.525177 18220 raft_consensus.cc:359] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.525838 18220 raft_consensus.cc:385] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.525919 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7635c52883ec4434bad0cc15f8733f72" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:54.526113 18220 raft_consensus.cc:740] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.526916 17818 raft_consensus.cc:3060] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.527359 19375 consensus_queue.cc:1048] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:54.551677 19403 raft_consensus.cc:493] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.552228 19403 raft_consensus.cc:515] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.553057 17818 raft_consensus.cc:2468] T 7635c52883ec4434bad0cc15f8733f72 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:54.555117 19403 leader_election.cc:290] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:54.555215 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7a1887d396f40a1a0c153dd0b2fe307" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:54.557545 18050 leader_election.cc:343] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.564471 19482 consensus_queue.cc:1048] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:54.584185 18224 raft_consensus.cc:359] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.586220 18224 raft_consensus.cc:385] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.586858 19529 consensus_queue.cc:1048] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:54.587247 18224 raft_consensus.cc:740] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.590055 18224 consensus_queue.cc:260] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.588881 18051 leader_election.cc:304] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:54.600597 19504 raft_consensus.cc:2804] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:54.601234 19504 raft_consensus.cc:697] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:54.602190 19504 consensus_queue.cc:237] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.526744 18220 consensus_queue.cc:260] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.611294 18220 ts_tablet_manager.cc:1434] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.088s user 0.008s sys 0.000s
I20260924 19:57:54.613817 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7a1887d396f40a1a0c153dd0b2fe307" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:54.614625 17817 raft_consensus.cc:2468] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:54.616146 18051 leader_election.cc:304] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:54.619006 19529 raft_consensus.cc:2804] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:54.619485 19529 raft_consensus.cc:493] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:54.619931 19529 raft_consensus.cc:3060] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.622573 18224 ts_tablet_manager.cc:1434] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.104s user 0.005s sys 0.003s
I20260924 19:57:54.625945 18223 tablet_bootstrap.cc:492] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:54.626380 18223 ts_tablet_manager.cc:1403] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.198s user 0.012s sys 0.013s
I20260924 19:57:54.628088 18224 tablet_bootstrap.cc:492] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:54.591679 18113 raft_consensus.cc:2468] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f6e3a7b06148918d230c77ab0c8202 in term 1.
I20260924 19:57:54.662613 19472 consensus_queue.cc:1048] T 2a1ac7e1b192465696640c02898bc39f P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:54.666324 18224 tablet_bootstrap.cc:654] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.689648 18224 tablet_bootstrap.cc:492] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:54.690035 18220 tablet_bootstrap.cc:492] T 7635c52883ec4434bad0cc15f8733f72 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:54.690196 18224 ts_tablet_manager.cc:1403] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.001s
I20260924 19:57:54.692878 18224 raft_consensus.cc:359] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:54.694507 18224 raft_consensus.cc:385] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.695619 18224 raft_consensus.cc:740] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.647373 18223 raft_consensus.cc:359] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.698652 18224 consensus_queue.cc:260] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:54.708604 18224 ts_tablet_manager.cc:1434] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20260924 19:57:54.719656 18224 tablet_bootstrap.cc:492] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202: Bootstrap starting.
I20260924 19:57:54.700984 18223 raft_consensus.cc:385] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.729637 18224 tablet_bootstrap.cc:654] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.729635 19529 raft_consensus.cc:515] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.734937 19557 raft_consensus.cc:493] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.739439 19557 raft_consensus.cc:515] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:54.741597 19557 leader_election.cc:290] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:54.723609 18223 raft_consensus.cc:740] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.744192 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7a1887d396f40a1a0c153dd0b2fe307" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:54.743743 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bea1ebdf647e4ef9a131a0398672b8ab" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:54.749629 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bea1ebdf647e4ef9a131a0398672b8ab" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:54.751329 17753 leader_election.cc:343] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:54.752406 17753 leader_election.cc:343] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:57:54.752622 18050 leader_election.cc:343] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.752794 17753 leader_election.cc:304] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:54.754699 19472 raft_consensus.cc:2749] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:54.767561 18220 tablet_bootstrap.cc:654] T 7635c52883ec4434bad0cc15f8733f72 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.781090 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7a1887d396f40a1a0c153dd0b2fe307" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:54.781684 17816 raft_consensus.cc:3060] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.787910 19557 raft_consensus.cc:493] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.788415 19557 raft_consensus.cc:515] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.790480 19557 leader_election.cc:290] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:54.796507 17816 raft_consensus.cc:2468] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:54.797504 18051 leader_election.cc:304] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:54.799806 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fe8d2d06dd4f83bef8c52a82693344" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:54.773537 18025 tablet.cc:2433] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:57:54.803822 17753 leader_election.cc:343] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.804531 19403 raft_consensus.cc:2804] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:54.805011 19403 raft_consensus.cc:697] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:54.807478 19582 consensus_queue.cc:1048] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:54.806331 19403 consensus_queue.cc:237] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.798228 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fe8d2d06dd4f83bef8c52a82693344" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:54.811782 17753 leader_election.cc:343] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.812206 17753 leader_election.cc:304] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:54.812903 19529 leader_election.cc:290] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:54.813344 19557 raft_consensus.cc:2749] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:54.821519 19557 raft_consensus.cc:493] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.822028 19557 raft_consensus.cc:515] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.824023 19557 leader_election.cc:290] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:54.828430 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70fea649583648ca86b4b3673ab0305f" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:54.830102 17753 leader_election.cc:343] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.831687 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70fea649583648ca86b4b3673ab0305f" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
W20260924 19:57:54.833146 17753 leader_election.cc:343] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.833513 17753 leader_election.cc:304] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:54.834470 19557 raft_consensus.cc:2749] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:54.840274 19557 raft_consensus.cc:493] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.840910 19557 raft_consensus.cc:515] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:54.843014 19557 leader_election.cc:290] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:54.844533 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00e3626ea154dc5a959f0fbb4e849dc" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
W20260924 19:57:54.846421 17753 leader_election.cc:343] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.846370 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00e3626ea154dc5a959f0fbb4e849dc" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:54.847839 18224 tablet_bootstrap.cc:492] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202: No bootstrap required, opened a new log
I20260924 19:57:54.848248 18224 ts_tablet_manager.cc:1403] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202: Time spent bootstrapping tablet: real 0.129s user 0.006s sys 0.007s
W20260924 19:57:54.849632 17753 leader_election.cc:343] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:54.849998 17753 leader_election.cc:304] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:54.850840 19557 raft_consensus.cc:2749] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:54.805238 18223 consensus_queue.cc:260] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.859184 19375 raft_consensus.cc:493] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.859618 19375 raft_consensus.cc:515] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.860435 18223 ts_tablet_manager.cc:1434] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.234s user 0.011s sys 0.014s
I20260924 19:57:54.861308 18223 tablet_bootstrap.cc:492] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:54.861986 19375 leader_election.cc:290] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:54.852962 19582 raft_consensus.cc:493] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.864494 19582 raft_consensus.cc:515] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.868340 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf025253c49a4c23b79a25333cad160e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:54.868925 17819 raft_consensus.cc:2468] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:54.869915 18051 leader_election.cc:304] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
W20260924 19:57:54.870604 17905 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 8aef0359a77240679240823612507ee8
I20260924 19:57:54.870802 19375 raft_consensus.cc:2804] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:54.871264 19375 raft_consensus.cc:493] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 19:57:54.871428 17906 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 8aef0359a77240679240823612507ee8
I20260924 19:57:54.871595 19375 raft_consensus.cc:3060] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.871803 17906 leader_election.cc:304] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:54.878158 19537 raft_consensus.cc:2749] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:54.878991 19375 raft_consensus.cc:515] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.879705 19594 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:54.880839 19594 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:54.882205 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf025253c49a4c23b79a25333cad160e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:54.882736 17819 raft_consensus.cc:3060] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.885076 17536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:54.887440 19594 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:54.889170 19582 leader_election.cc:290] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:54.889714 17819 raft_consensus.cc:2468] T bf025253c49a4c23b79a25333cad160e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:54.892529 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf025253c49a4c23b79a25333cad160e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:54.893375 18051 leader_election.cc:304] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:54.894080 17969 raft_consensus.cc:2468] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:54.897943 19504 raft_consensus.cc:2804] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:54.899258 19504 raft_consensus.cc:697] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:54.899712 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf025253c49a4c23b79a25333cad160e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:54.900141 19504 consensus_queue.cc:237] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.900658 17969 raft_consensus.cc:3060] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:54.913725 19375 leader_election.cc:290] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:54.915632 18224 raft_consensus.cc:359] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.924484 18224 raft_consensus.cc:385] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:54.924832 18224 raft_consensus.cc:740] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f6e3a7b06148918d230c77ab0c8202, State: Initialized, Role: FOLLOWER
I20260924 19:57:54.925926 18224 consensus_queue.cc:260] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.942641 18224 ts_tablet_manager.cc:1434] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202: Time spent starting tablet: real 0.094s user 0.005s sys 0.000s
I20260924 19:57:54.955468 18220 tablet_bootstrap.cc:492] T 7635c52883ec4434bad0cc15f8733f72 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:54.955938 18220 ts_tablet_manager.cc:1403] T 7635c52883ec4434bad0cc15f8733f72 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.266s user 0.012s sys 0.001s
I20260924 19:57:54.958328 18220 raft_consensus.cc:359] T 7635c52883ec4434bad0cc15f8733f72 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.962651 18223 tablet_bootstrap.cc:654] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:54.991328 19557 raft_consensus.cc:493] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:54.991881 19557 raft_consensus.cc:515] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:54.993876 19557 leader_election.cc:290] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:55.015906 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d25793b65da4315b6f95726374ea46c" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:54.958961 18220 raft_consensus.cc:385] T 7635c52883ec4434bad0cc15f8733f72 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:55.028262 18220 raft_consensus.cc:740] T 7635c52883ec4434bad0cc15f8733f72 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:55.029551 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d25793b65da4315b6f95726374ea46c" candidate_uuid: "c4f6e3a7b06148918d230c77ab0c8202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:55.056612 19557 consensus_queue.cc:1048] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:55.059304 17753 leader_election.cc:343] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:55.067587 18220 consensus_queue.cc:260] T 7635c52883ec4434bad0cc15f8733f72 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:55.070657 17753 leader_election.cc:343] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:55.082068 18220 ts_tablet_manager.cc:1434] T 7635c52883ec4434bad0cc15f8733f72 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.126s user 0.006s sys 0.001s
I20260924 19:57:55.083012 18220 tablet_bootstrap.cc:492] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:55.087991 17969 raft_consensus.cc:2468] T bf025253c49a4c23b79a25333cad160e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:55.092849 17753 leader_election.cc:304] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202; no voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:55.093719 19557 raft_consensus.cc:2749] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:55.151913 18223 tablet_bootstrap.cc:492] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:55.152535 18223 ts_tablet_manager.cc:1403] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.291s user 0.003s sys 0.012s
I20260924 19:57:55.155366 18223 raft_consensus.cc:359] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.156119 18223 raft_consensus.cc:385] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:55.156436 18223 raft_consensus.cc:740] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:55.179826 19542 consensus_queue.cc:1048] T c058a49bfd5544b58b76edfd3fadbd1a P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:55.189116 18220 tablet_bootstrap.cc:654] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:55.195518 19403 raft_consensus.cc:493] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:55.235844 19403 raft_consensus.cc:515] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.238346 19403 leader_election.cc:290] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:55.239353 18223 consensus_queue.cc:260] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.242058 18223 ts_tablet_manager.cc:1434] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.089s user 0.005s sys 0.000s
I20260924 19:57:55.247288 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f116f866c2b4674b4eef531cf00af66" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:55.247874 17816 raft_consensus.cc:2468] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:55.250475 18051 leader_election.cc:304] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:55.251542 19403 raft_consensus.cc:2804] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:55.251958 19403 raft_consensus.cc:493] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:55.252341 19403 raft_consensus.cc:3060] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:55.296962 18223 tablet_bootstrap.cc:492] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:55.321473 18220 tablet_bootstrap.cc:492] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:55.322024 18220 ts_tablet_manager.cc:1403] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.239s user 0.007s sys 0.008s
I20260924 19:57:55.324609 18220 raft_consensus.cc:359] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.325357 18220 raft_consensus.cc:385] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:55.325678 18220 raft_consensus.cc:740] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:55.326681 18220 consensus_queue.cc:260] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.328150 19403 raft_consensus.cc:515] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.330627 19403 leader_election.cc:290] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:55.339564 17816 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f116f866c2b4674b4eef531cf00af66" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:55.340137 17816 raft_consensus.cc:3060] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:55.347354 17816 raft_consensus.cc:2468] T 5f116f866c2b4674b4eef531cf00af66 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:55.348526 18051 leader_election.cc:304] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:55.351714 19529 raft_consensus.cc:2804] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:55.352177 19529 raft_consensus.cc:697] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:55.352936 19529 consensus_queue.cc:237] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.353771 18223 tablet_bootstrap.cc:654] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:55.341737 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f116f866c2b4674b4eef531cf00af66" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:55.380074 18220 ts_tablet_manager.cc:1434] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.058s user 0.006s sys 0.000s
I20260924 19:57:55.400153 18220 tablet_bootstrap.cc:492] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:55.422878 19472 consensus_queue.cc:1048] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:55.441759 17969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f116f866c2b4674b4eef531cf00af66" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:55.445755 17969 raft_consensus.cc:3060] T 7635c52883ec4434bad0cc15f8733f72 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:55.449450 18220 tablet_bootstrap.cc:654] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:55.465687 18050 leader_election.cc:343] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:55.470160 18220 tablet_bootstrap.cc:492] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:55.470599 18220 ts_tablet_manager.cc:1403] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.006s
I20260924 19:57:55.473104 18220 raft_consensus.cc:359] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.473786 18220 raft_consensus.cc:385] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:55.474058 18220 raft_consensus.cc:740] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
W20260924 19:57:55.475910 18903 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.585s (SENT) [suppressed 3 similar messages]
I20260924 19:57:55.474717 18220 consensus_queue.cc:260] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.479087 19472 consensus_queue.cc:1048] T 8c05ec2ba01448d998c6ffa1525de984 P c4f6e3a7b06148918d230c77ab0c8202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:55.488351 18223 tablet_bootstrap.cc:492] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:55.499773 18220 ts_tablet_manager.cc:1434] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260924 19:57:55.500526 18223 ts_tablet_manager.cc:1403] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.209s user 0.007s sys 0.005s
I20260924 19:57:55.513527 18220 tablet_bootstrap.cc:492] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
W20260924 19:57:55.516173 18050 leader_election.cc:343] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:55.518764 19529 consensus_queue.cc:1048] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:55.543650 18223 raft_consensus.cc:359] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.544525 18223 raft_consensus.cc:385] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:55.544772 18223 raft_consensus.cc:740] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:55.545429 18223 consensus_queue.cc:260] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.558899 18223 ts_tablet_manager.cc:1434] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.058s user 0.001s sys 0.005s
I20260924 19:57:55.559798 18223 tablet_bootstrap.cc:492] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:55.580063 18223 tablet_bootstrap.cc:654] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:55.603940 18223 tablet_bootstrap.cc:492] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:55.617615 18220 tablet_bootstrap.cc:654] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:55.620299 19529 consensus_queue.cc:1048] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:55.604333 18223 ts_tablet_manager.cc:1403] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.000s
I20260924 19:57:55.650578 18223 raft_consensus.cc:359] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.676951 18220 tablet_bootstrap.cc:492] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:55.677392 18220 ts_tablet_manager.cc:1403] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.165s user 0.009s sys 0.001s
I20260924 19:57:55.680102 18220 raft_consensus.cc:359] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.680725 18220 raft_consensus.cc:385] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:55.681069 18220 raft_consensus.cc:740] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:55.681650 18220 consensus_queue.cc:260] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.683615 18220 ts_tablet_manager.cc:1434] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 19:57:55.684458 18220 tablet_bootstrap.cc:492] T 57102b1727e94a90981509c82c082441 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:55.692839 18223 raft_consensus.cc:385] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:55.693153 18223 raft_consensus.cc:740] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:55.704205 18223 consensus_queue.cc:260] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.706709 18223 ts_tablet_manager.cc:1434] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.066s user 0.005s sys 0.000s
I20260924 19:57:55.725081 18223 tablet_bootstrap.cc:492] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:55.752669 18220 tablet_bootstrap.cc:654] T 57102b1727e94a90981509c82c082441 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:55.792973 18220 tablet_bootstrap.cc:492] T 57102b1727e94a90981509c82c082441 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:55.793469 18220 ts_tablet_manager.cc:1403] T 57102b1727e94a90981509c82c082441 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.008s
I20260924 19:57:55.795938 18220 raft_consensus.cc:359] T 57102b1727e94a90981509c82c082441 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.796568 18220 raft_consensus.cc:385] T 57102b1727e94a90981509c82c082441 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:55.796849 18220 raft_consensus.cc:740] T 57102b1727e94a90981509c82c082441 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:55.797510 18220 consensus_queue.cc:260] T 57102b1727e94a90981509c82c082441 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.812376 18220 ts_tablet_manager.cc:1434] T 57102b1727e94a90981509c82c082441 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20260924 19:57:55.822788 18220 tablet_bootstrap.cc:492] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:55.845873 19616 raft_consensus.cc:493] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:55.846392 19616 raft_consensus.cc:515] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.848500 19616 leader_election.cc:290] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:55.849400 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7a1887d396f40a1a0c153dd0b2fe307" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:55.850134 17819 raft_consensus.cc:2393] T b7a1887d396f40a1a0c153dd0b2fe307 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f74e6ed351284d7fb110e44336ce643a in current term 1: Already voted for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in this term.
I20260924 19:57:55.741678 18223 tablet_bootstrap.cc:654] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:55.834971 19375 consensus_queue.cc:1048] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.046s
I20260924 19:57:55.778208 19403 consensus_queue.cc:1048] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.115s
I20260924 19:57:55.829651 18220 tablet_bootstrap.cc:654] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:55.768468 19500 raft_consensus.cc:493] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:55.914132 19626 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:55.914680 19626 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:55.918406 18220 tablet_bootstrap.cc:492] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:55.918859 18220 ts_tablet_manager.cc:1403] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.096s user 0.006s sys 0.003s
I20260924 19:57:55.921146 19500 raft_consensus.cc:515] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.921434 18220 raft_consensus.cc:359] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:55.922029 18220 raft_consensus.cc:385] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:55.934412 18939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:55.936518 19626 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:55.935729 18223 tablet_bootstrap.cc:492] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:55.937958 18223 ts_tablet_manager.cc:1403] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.213s user 0.005s sys 0.008s
I20260924 19:57:55.960309 18223 raft_consensus.cc:359] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.961012 18223 raft_consensus.cc:385] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:55.964588 17812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb111f7a189c4f00a41686d647c64e65" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:55.965303 17812 raft_consensus.cc:2468] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:55.966312 18051 leader_election.cc:304] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:55.922308 18220 raft_consensus.cc:740] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:55.967723 19375 raft_consensus.cc:2804] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:55.967592 18220 consensus_queue.cc:260] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:55.983196 19375 raft_consensus.cc:493] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:55.983819 18220 ts_tablet_manager.cc:1434] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.065s user 0.008s sys 0.000s
I20260924 19:57:55.968191 18223 raft_consensus.cc:740] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:55.984169 19375 raft_consensus.cc:3060] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:55.984637 18223 consensus_queue.cc:260] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:55.986083 18223 ts_tablet_manager.cc:1434] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.048s user 0.000s sys 0.005s
I20260924 19:57:55.979768 19500 leader_election.cc:290] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.026268 18223 tablet_bootstrap.cc:492] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:56.048599 17969 raft_consensus.cc:3060] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.062907 18223 tablet_bootstrap.cc:654] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:56.063030 19529 consensus_queue.cc:1048] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:56.069977 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb111f7a189c4f00a41686d647c64e65" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:56.086519 18220 tablet_bootstrap.cc:492] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:56.092370 18220 tablet_bootstrap.cc:654] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:56.097172 18903 leader_election.cc:336] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.585s (SENT)
I20260924 19:57:56.097612 18903 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:56.108968 18223 tablet_bootstrap.cc:492] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:56.109385 18223 ts_tablet_manager.cc:1403] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.083s user 0.009s sys 0.000s
I20260924 19:57:56.121894 18220 tablet_bootstrap.cc:492] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:56.122310 18220 ts_tablet_manager.cc:1403] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.000s
I20260924 19:57:56.123458 19633 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 19:57:56.125088 18220 raft_consensus.cc:359] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.125713 18220 raft_consensus.cc:385] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:56.125974 18220 raft_consensus.cc:740] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:56.127837 18223 raft_consensus.cc:359] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.131388 18223 raft_consensus.cc:385] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:56.131675 18223 raft_consensus.cc:740] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:56.139097 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7a1887d396f40a1a0c153dd0b2fe307" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:56.144703 19500 consensus_queue.cc:1048] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:56.148361 19375 raft_consensus.cc:515] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.157171 17812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb111f7a189c4f00a41686d647c64e65" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:56.157830 17812 raft_consensus.cc:3060] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.168047 19375 leader_election.cc:290] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.195043 18223 consensus_queue.cc:260] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.203657 18223 ts_tablet_manager.cc:1434] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.094s user 0.007s sys 0.000s
I20260924 19:57:56.126551 18220 consensus_queue.cc:260] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.205394 18220 ts_tablet_manager.cc:1434] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.083s user 0.006s sys 0.000s
I20260924 19:57:56.206276 18220 tablet_bootstrap.cc:492] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:56.211826 18223 tablet_bootstrap.cc:492] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:56.218242 19403 consensus_queue.cc:1048] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:56.246443 18223 tablet_bootstrap.cc:654] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:56.258308 18220 tablet_bootstrap.cc:654] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:56.260572 19616 raft_consensus.cc:493] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:56.261089 19616 raft_consensus.cc:515] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.263237 19616 leader_election.cc:290] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.264448 19529 raft_consensus.cc:493] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:56.264925 19529 raft_consensus.cc:515] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.266739 19529 leader_election.cc:290] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:56.268306 17906 leader_election.cc:343] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 44199d2655f14dcf97afe2180789aae4
W20260924 19:57:56.269114 18050 leader_election.cc:343] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:56.273415 19500 consensus_queue.cc:1048] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:56.164767 17812 raft_consensus.cc:2468] T cb111f7a189c4f00a41686d647c64e65 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:56.275075 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57102b1727e94a90981509c82c082441" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:56.275705 17819 raft_consensus.cc:2468] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:56.277307 18051 leader_election.cc:304] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:56.278556 19529 raft_consensus.cc:2804] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:56.279018 19529 raft_consensus.cc:697] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:56.280004 19529 consensus_queue.cc:237] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.285749 18051 leader_election.cc:304] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:56.287139 19500 raft_consensus.cc:2804] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:56.287519 19500 raft_consensus.cc:493] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:56.287906 19500 raft_consensus.cc:3060] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.301967 19500 raft_consensus.cc:515] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.304571 19500 leader_election.cc:290] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.310428 18223 tablet_bootstrap.cc:492] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:56.310827 18223 ts_tablet_manager.cc:1403] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.099s user 0.009s sys 0.000s
I20260924 19:57:56.313351 18223 raft_consensus.cc:359] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.314050 18223 raft_consensus.cc:385] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:56.314321 18223 raft_consensus.cc:740] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:56.314973 18223 consensus_queue.cc:260] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.316946 18223 ts_tablet_manager.cc:1434] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260924 19:57:56.317842 18223 tablet_bootstrap.cc:492] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:56.347445 18220 tablet_bootstrap.cc:492] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:56.351425 18220 ts_tablet_manager.cc:1403] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.145s user 0.010s sys 0.001s
I20260924 19:57:56.354688 18223 tablet_bootstrap.cc:654] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:56.356722 18220 raft_consensus.cc:359] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.362323 18220 raft_consensus.cc:385] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:56.362675 18220 raft_consensus.cc:740] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:56.363421 18220 consensus_queue.cc:260] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.366060 18220 ts_tablet_manager.cc:1434] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20260924 19:57:56.368492 17905 leader_election.cc:304] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:56.371196 19403 consensus_queue.cc:1048] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:56.374670 19644 raft_consensus.cc:2749] T b7a1887d396f40a1a0c153dd0b2fe307 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:56.378679 19500 raft_consensus.cc:493] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:56.379283 19500 raft_consensus.cc:515] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.380715 19615 raft_consensus.cc:493] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:56.381156 19615 raft_consensus.cc:515] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.381539 19500 leader_election.cc:290] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.382441 19643 raft_consensus.cc:493] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:56.382863 19643 raft_consensus.cc:515] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.383100 19615 leader_election.cc:290] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.397192 18220 tablet_bootstrap.cc:492] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:56.403928 19643 leader_election.cc:290] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.416494 19643 raft_consensus.cc:493] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:56.424785 17967 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb111f7a189c4f00a41686d647c64e65" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:56.426015 18223 tablet_bootstrap.cc:492] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:56.426455 18223 ts_tablet_manager.cc:1403] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.109s user 0.015s sys 0.010s
I20260924 19:57:56.428478 17967 raft_consensus.cc:3060] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.428979 18223 raft_consensus.cc:359] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.429693 18223 raft_consensus.cc:385] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:56.429965 18223 raft_consensus.cc:740] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:56.416955 19643 raft_consensus.cc:515] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.430645 18223 consensus_queue.cc:260] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.432152 18223 ts_tablet_manager.cc:1434] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 19:57:56.438506 18220 tablet_bootstrap.cc:654] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:56.447496 19403 raft_consensus.cc:493] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:56.448019 19403 raft_consensus.cc:515] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.439356 19643 leader_election.cc:290] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:56.446866 17971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57102b1727e94a90981509c82c082441" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:56.486474 17971 raft_consensus.cc:3060] T 57102b1727e94a90981509c82c082441 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.449425 17966 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57102b1727e94a90981509c82c082441" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:56.464780 17968 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8523125e35554830aff04b17677d5ceb" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:56.492470 17968 raft_consensus.cc:2468] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:56.472265 17963 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd5915ee98a4eda94479e9330cf8d04" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:56.493974 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f116f866c2b4674b4eef531cf00af66" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:56.494975 17971 raft_consensus.cc:2468] T 57102b1727e94a90981509c82c082441 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:56.497118 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2de3877b94b44c94b6d0fdad93f8aa79" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:56.497619 18113 raft_consensus.cc:2468] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:56.453363 19403 leader_election.cc:290] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:56.509683 18050 leader_election.cc:343] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:56.511761 19500 consensus_queue.cc:1048] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:56.515615 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bea1ebdf647e4ef9a131a0398672b8ab" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:56.517982 18223 tablet_bootstrap.cc:492] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:56.524829 18223 tablet_bootstrap.cc:654] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:56.525166 18220 tablet_bootstrap.cc:492] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:56.525616 18220 ts_tablet_manager.cc:1403] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.129s user 0.005s sys 0.003s
W20260924 19:57:56.526827 17905 leader_election.cc:343] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 44199d2655f14dcf97afe2180789aae4
I20260924 19:57:56.527303 17905 leader_election.cc:304] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:56.527976 18220 raft_consensus.cc:359] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.528570 18220 raft_consensus.cc:385] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:56.528823 18220 raft_consensus.cc:740] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:56.529119 19643 raft_consensus.cc:2749] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:56.529778 18220 consensus_queue.cc:260] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.544833 18220 ts_tablet_manager.cc:1434] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20260924 19:57:56.545846 17905 leader_election.cc:304] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:56.551371 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2de3877b94b44c94b6d0fdad93f8aa79" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:56.552732 17810 raft_consensus.cc:2468] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:56.551234 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f116f866c2b4674b4eef531cf00af66" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:56.556622 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bea1ebdf647e4ef9a131a0398672b8ab" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:56.557408 17818 raft_consensus.cc:2468] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:56.559645 19641 raft_consensus.cc:2804] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:56.559998 19641 raft_consensus.cc:493] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:56.560324 19641 raft_consensus.cc:3060] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.560825 17906 leader_election.cc:304] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:56.561911 17906 leader_election.cc:304] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:56.562788 19644 raft_consensus.cc:2804] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:56.562837 19637 raft_consensus.cc:2749] T 5f116f866c2b4674b4eef531cf00af66 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:56.563551 19644 raft_consensus.cc:493] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:56.563907 19644 raft_consensus.cc:3060] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.568775 19641 raft_consensus.cc:515] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:56.571331 17905 leader_election.cc:343] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:56.570737 19644 raft_consensus.cc:515] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.570883 19641 leader_election.cc:290] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.574991 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bea1ebdf647e4ef9a131a0398672b8ab" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:56.575744 17818 raft_consensus.cc:3060] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.575874 19644 leader_election.cc:290] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:56.575990 18050 leader_election.cc:304] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
W20260924 19:57:56.576733 18050 leader_election.cc:343] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:56.577517 18050 leader_election.cc:304] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:56.578413 19504 raft_consensus.cc:2804] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:56.578835 19504 raft_consensus.cc:697] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:56.579643 19504 consensus_queue.cc:237] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.577766 19500 raft_consensus.cc:2804] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:56.584136 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2de3877b94b44c94b6d0fdad93f8aa79" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:56.584401 19500 raft_consensus.cc:493] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:56.584774 19500 raft_consensus.cc:3060] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.584853 17818 raft_consensus.cc:2468] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:56.585362 18220 tablet_bootstrap.cc:492] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:56.590915 18220 tablet_bootstrap.cc:654] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:56.584913 17819 raft_consensus.cc:3060] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.596369 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2de3877b94b44c94b6d0fdad93f8aa79" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:56.597005 18113 raft_consensus.cc:3060] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.592031 17906 leader_election.cc:304] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:56.603101 19644 raft_consensus.cc:2804] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:56.603650 19644 raft_consensus.cc:697] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:56.603761 19643 raft_consensus.cc:493] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:56.604203 19643 raft_consensus.cc:515] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.606177 19643 leader_election.cc:290] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.604593 19644 consensus_queue.cc:237] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.613940 18113 raft_consensus.cc:2468] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:56.625443 17819 raft_consensus.cc:2468] T 2de3877b94b44c94b6d0fdad93f8aa79 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:56.628376 18220 tablet_bootstrap.cc:492] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:56.630868 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bea1ebdf647e4ef9a131a0398672b8ab" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:56.632804 18223 tablet_bootstrap.cc:492] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:56.633409 18223 ts_tablet_manager.cc:1403] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.116s user 0.012s sys 0.003s
I20260924 19:57:56.633967 18220 ts_tablet_manager.cc:1403] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.000s
I20260924 19:57:56.640770 18223 raft_consensus.cc:359] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.641391 18223 raft_consensus.cc:385] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:56.641691 18223 raft_consensus.cc:740] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:56.642369 18223 consensus_queue.cc:260] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.644382 18223 ts_tablet_manager.cc:1434] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.011s user 0.003s sys 0.002s
I20260924 19:57:56.645335 18223 tablet_bootstrap.cc:492] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:56.624267 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8523125e35554830aff04b17677d5ceb" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:56.650900 18223 tablet_bootstrap.cc:654] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:56.626416 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57102b1727e94a90981509c82c082441" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:56.653283 18220 raft_consensus.cc:359] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.655961 18220 raft_consensus.cc:385] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:56.656761 18220 raft_consensus.cc:740] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:56.658025 18220 consensus_queue.cc:260] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.661731 18223 tablet_bootstrap.cc:492] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:56.662194 18223 ts_tablet_manager.cc:1403] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20260924 19:57:56.663431 18220 ts_tablet_manager.cc:1434] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
I20260924 19:57:56.664803 18220 tablet_bootstrap.cc:492] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:56.664479 18223 raft_consensus.cc:359] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.665067 18223 raft_consensus.cc:385] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:56.665323 18223 raft_consensus.cc:740] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:56.666601 18223 consensus_queue.cc:260] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.668735 18223 ts_tablet_manager.cc:1434] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 19:57:56.658855 19500 raft_consensus.cc:515] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.678206 17906 leader_election.cc:304] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
W20260924 19:57:56.680509 17905 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
W20260924 19:57:56.681128 17905 leader_election.cc:343] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:56.683326 17818 raft_consensus.cc:3060] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.685081 18223 tablet_bootstrap.cc:492] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:56.687330 17817 raft_consensus.cc:2468] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:56.688910 19500 leader_election.cc:290] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:56.695330 17521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.188:38761, user_credentials={real_user=slave}} blocked reactor thread for 44717.5us
I20260924 19:57:56.697921 19641 raft_consensus.cc:2804] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:56.698414 19641 raft_consensus.cc:697] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:56.694869 17809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8523125e35554830aff04b17677d5ceb" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:56.699378 17809 raft_consensus.cc:3060] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.699255 19641 consensus_queue.cc:237] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.703362 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8523125e35554830aff04b17677d5ceb" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:56.703858 17970 raft_consensus.cc:3060] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.709909 17970 raft_consensus.cc:2468] T 8523125e35554830aff04b17677d5ceb P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:56.715662 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd5915ee98a4eda94479e9330cf8d04" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:56.716499 17810 raft_consensus.cc:2468] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:56.730914 18050 leader_election.cc:304] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:56.731689 19504 raft_consensus.cc:2804] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:56.732152 19504 raft_consensus.cc:697] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:56.732999 19504 consensus_queue.cc:237] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.741255 18051 leader_election.cc:304] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:56.742077 19375 raft_consensus.cc:2804] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:56.742394 19375 raft_consensus.cc:493] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:56.742735 19375 raft_consensus.cc:3060] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.747335 17818 raft_consensus.cc:2468] T 57102b1727e94a90981509c82c082441 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:56.759408 19615 raft_consensus.cc:493] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:56.759896 19615 raft_consensus.cc:515] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.761796 19615 leader_election.cc:290] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:56.766551 18113 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "144355b18c2f413baa28dd632623db52" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:56.767110 18113 raft_consensus.cc:2468] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:56.769654 18223 tablet_bootstrap.cc:654] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:56.783633 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "144355b18c2f413baa28dd632623db52" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:56.784282 17819 raft_consensus.cc:2468] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:56.786569 17906 leader_election.cc:304] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:56.787249 19643 raft_consensus.cc:2804] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:56.787609 19643 raft_consensus.cc:493] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:56.787921 19643 raft_consensus.cc:3060] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.790686 19375 raft_consensus.cc:515] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.796929 17809 raft_consensus.cc:2468] T 8523125e35554830aff04b17677d5ceb P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
W20260924 19:57:56.798714 17906 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:57:56.799218 17906 leader_election.cc:304] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:56.800208 19615 raft_consensus.cc:2749] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:56.805874 19643 raft_consensus.cc:515] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.807868 19643 leader_election.cc:290] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:56.808671 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "144355b18c2f413baa28dd632623db52" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:56.809367 17818 raft_consensus.cc:3060] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.825589 18220 tablet_bootstrap.cc:654] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:56.834815 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "144355b18c2f413baa28dd632623db52" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:56.835469 18112 raft_consensus.cc:3060] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.837248 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd5915ee98a4eda94479e9330cf8d04" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
W20260924 19:57:56.841387 18050 leader_election.cc:343] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:57:56.845665 18112 raft_consensus.cc:2468] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:56.848786 17905 leader_election.cc:304] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:56.849720 19375 leader_election.cc:290] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.855463 19641 raft_consensus.cc:2804] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:56.856370 19641 raft_consensus.cc:697] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:56.857182 19641 consensus_queue.cc:237] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.868960 17818 raft_consensus.cc:2468] T 144355b18c2f413baa28dd632623db52 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
W20260924 19:57:56.895648 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 46943.1us
I20260924 19:57:56.897373 19615 raft_consensus.cc:493] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:56.897835 19615 raft_consensus.cc:515] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.900662 19615 leader_election.cc:290] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.915053 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18bb8f3d43df4aae8614b6dba9ffc668" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:56.917004 18112 raft_consensus.cc:2468] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:56.920948 18223 tablet_bootstrap.cc:492] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:56.921397 18223 ts_tablet_manager.cc:1403] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.237s user 0.006s sys 0.003s
I20260924 19:57:56.877568 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd5915ee98a4eda94479e9330cf8d04" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:56.902173 17817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18bb8f3d43df4aae8614b6dba9ffc668" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:56.926170 17817 raft_consensus.cc:2468] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:56.927139 17819 raft_consensus.cc:3060] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.930888 17905 leader_election.cc:304] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:56.931586 19641 raft_consensus.cc:2804] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:56.931927 19641 raft_consensus.cc:493] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:56.932237 19641 raft_consensus.cc:3060] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.934475 17819 raft_consensus.cc:2468] T fbd5915ee98a4eda94479e9330cf8d04 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
W20260924 19:57:56.937932 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 54581.2us
I20260924 19:57:56.939437 18051 leader_election.cc:304] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters: f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:56.940805 18223 raft_consensus.cc:359] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.941480 18223 raft_consensus.cc:385] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:56.941740 18223 raft_consensus.cc:740] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:56.942427 18223 consensus_queue.cc:260] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:56.943104 18220 tablet_bootstrap.cc:492] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:56.943682 18220 ts_tablet_manager.cc:1403] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.279s user 0.008s sys 0.003s
I20260924 19:57:56.946291 19500 raft_consensus.cc:2804] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:56.946715 19500 raft_consensus.cc:697] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:56.947501 19500 consensus_queue.cc:237] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.948052 19504 raft_consensus.cc:493] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:56.948442 19504 raft_consensus.cc:515] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.950261 19504 leader_election.cc:290] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:56.954604 18220 raft_consensus.cc:359] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.957190 18220 raft_consensus.cc:385] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:56.957445 18220 raft_consensus.cc:740] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:56.958040 18220 consensus_queue.cc:260] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:56.964978 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59e737a68dd443f9898113b2b2a401e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:56.965548 17970 raft_consensus.cc:2468] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:56.966485 18050 leader_election.cc:304] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:56.967243 19375 raft_consensus.cc:2804] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:56.967628 19375 raft_consensus.cc:493] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:56.967999 19375 raft_consensus.cc:3060] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:56.979076 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59e737a68dd443f9898113b2b2a401e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:56.981959 17818 raft_consensus.cc:2468] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:56.992362 18223 ts_tablet_manager.cc:1434] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.071s user 0.007s sys 0.000s
I20260924 19:57:56.993608 18223 tablet_bootstrap.cc:492] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:56.993299 18220 ts_tablet_manager.cc:1434] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20260924 19:57:57.003760 18220 tablet_bootstrap.cc:492] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:57.013689 19641 raft_consensus.cc:515] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
W20260924 19:57:57.031730 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 53768.2us
I20260924 19:57:57.040077 18220 tablet_bootstrap.cc:654] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:57.043680 19641 leader_election.cc:290] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:57.044775 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18bb8f3d43df4aae8614b6dba9ffc668" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:57.045288 17819 raft_consensus.cc:3060] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.045285 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18bb8f3d43df4aae8614b6dba9ffc668" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:57.045823 18112 raft_consensus.cc:3060] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.054870 18112 raft_consensus.cc:2468] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:57.058125 17905 leader_election.cc:304] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:57.058877 19641 raft_consensus.cc:2804] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:57.058890 17819 raft_consensus.cc:2468] T 18bb8f3d43df4aae8614b6dba9ffc668 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:57.059602 19641 raft_consensus.cc:697] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:57.060374 19641 consensus_queue.cc:237] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.065454 18220 tablet_bootstrap.cc:492] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:57.065864 18220 ts_tablet_manager.cc:1403] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.000s
I20260924 19:57:57.068257 18220 raft_consensus.cc:359] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.068876 18220 raft_consensus.cc:385] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:57.069136 18220 raft_consensus.cc:740] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:57.062991 18223 tablet_bootstrap.cc:654] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:57.103912 18220 consensus_queue.cc:260] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.113967 19375 raft_consensus.cc:515] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.116566 19375 leader_election.cc:290] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:57.116448 17970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59e737a68dd443f9898113b2b2a401e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:57.116954 17970 raft_consensus.cc:3060] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.118434 18223 tablet_bootstrap.cc:492] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:57.122660 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59e737a68dd443f9898113b2b2a401e" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:57.119453 18223 ts_tablet_manager.cc:1403] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.126s user 0.010s sys 0.000s
I20260924 19:57:57.123178 17819 raft_consensus.cc:3060] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.123296 18220 ts_tablet_manager.cc:1434] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.049s user 0.006s sys 0.001s
I20260924 19:57:57.134387 17970 raft_consensus.cc:2468] T e59e737a68dd443f9898113b2b2a401e P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:57.136058 18050 leader_election.cc:304] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:57.137351 19403 raft_consensus.cc:2804] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:57.137797 19403 raft_consensus.cc:697] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:57.139145 19403 consensus_queue.cc:237] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.132082 19375 raft_consensus.cc:493] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:57.144695 19500 raft_consensus.cc:493] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:57.146060 19500 raft_consensus.cc:515] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:57.154042 17965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bea1ebdf647e4ef9a131a0398672b8ab" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:57.155390 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bea1ebdf647e4ef9a131a0398672b8ab" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:57.156205 17818 raft_consensus.cc:2393] T bea1ebdf647e4ef9a131a0398672b8ab P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eaf64bfb64aa42948c9bc1411ce4ce35 in current term 1: Already voted for candidate f74e6ed351284d7fb110e44336ce643a in this term.
I20260924 19:57:57.159646 19677 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:57.160138 19677 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:57.162949 17536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:57:57.158684 19500 leader_election.cc:290] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:57.168401 18220 tablet_bootstrap.cc:492] T fbd5915ee98a4eda94479e9330cf8d04 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:57.178746 19677 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:57.188263 18051 leader_election.cc:304] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35; no voters: c4f6e3a7b06148918d230c77ab0c8202, f74e6ed351284d7fb110e44336ce643a
I20260924 19:57:57.198030 19500 raft_consensus.cc:3060] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.200419 18223 raft_consensus.cc:359] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.201128 18223 raft_consensus.cc:385] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:57.201401 18223 raft_consensus.cc:740] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:57.212807 18223 consensus_queue.cc:260] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.218021 18223 ts_tablet_manager.cc:1434] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.094s user 0.006s sys 0.000s
I20260924 19:57:57.228905 18220 tablet_bootstrap.cc:654] T fbd5915ee98a4eda94479e9330cf8d04 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:57.237975 18223 tablet_bootstrap.cc:492] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:57:57.237799 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 51654.1us
I20260924 19:57:57.147540 19375 raft_consensus.cc:515] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.252216 17965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70fea649583648ca86b4b3673ab0305f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:57.252806 17965 raft_consensus.cc:2468] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:57.254215 18050 leader_election.cc:304] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:57.231659 17819 raft_consensus.cc:2468] T e59e737a68dd443f9898113b2b2a401e P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:57.255019 19672 raft_consensus.cc:2804] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:57.255621 19672 raft_consensus.cc:493] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:57.255950 19672 raft_consensus.cc:3060] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.267491 19672 raft_consensus.cc:515] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.271350 17965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70fea649583648ca86b4b3673ab0305f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:57.271759 19672 leader_election.cc:290] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:57.271881 17965 raft_consensus.cc:3060] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.284861 17965 raft_consensus.cc:2468] T 70fea649583648ca86b4b3673ab0305f P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:57.292609 18050 leader_election.cc:304] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:57.293424 19504 raft_consensus.cc:2804] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:57.251006 19375 leader_election.cc:290] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:57.293895 19504 raft_consensus.cc:697] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:57.296393 19504 consensus_queue.cc:237] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.303689 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70fea649583648ca86b4b3673ab0305f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:57.304347 17818 raft_consensus.cc:2468] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:57.314733 18223 tablet_bootstrap.cc:654] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:57.317306 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70fea649583648ca86b4b3673ab0305f" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:57.317862 17818 raft_consensus.cc:3060] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.333669 17965 raft_consensus.cc:3060] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.350260 18223 tablet_bootstrap.cc:492] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:57.350673 18223 ts_tablet_manager.cc:1403] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.113s user 0.011s sys 0.000s
I20260924 19:57:57.350818 19504 consensus_queue.cc:1048] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:57.352152 19500 raft_consensus.cc:2749] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:57.357632 18223 raft_consensus.cc:359] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.358287 18223 raft_consensus.cc:385] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:57.358567 18223 raft_consensus.cc:740] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:57.359228 18223 consensus_queue.cc:260] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.365036 18223 ts_tablet_manager.cc:1434] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.014s user 0.000s sys 0.007s
I20260924 19:57:57.374661 18220 tablet_bootstrap.cc:492] T fbd5915ee98a4eda94479e9330cf8d04 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:57.375262 18220 ts_tablet_manager.cc:1403] T fbd5915ee98a4eda94479e9330cf8d04 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.207s user 0.007s sys 0.006s
I20260924 19:57:57.378733 18220 raft_consensus.cc:359] T fbd5915ee98a4eda94479e9330cf8d04 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.380394 18220 raft_consensus.cc:385] T fbd5915ee98a4eda94479e9330cf8d04 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:57.380690 18220 raft_consensus.cc:740] T fbd5915ee98a4eda94479e9330cf8d04 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:57.381342 18220 consensus_queue.cc:260] T fbd5915ee98a4eda94479e9330cf8d04 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.392025 18223 tablet_bootstrap.cc:492] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:57.392894 18220 ts_tablet_manager.cc:1434] T fbd5915ee98a4eda94479e9330cf8d04 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.017s user 0.004s sys 0.001s
I20260924 19:57:57.393853 18220 tablet_bootstrap.cc:492] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:57.406560 18220 tablet_bootstrap.cc:654] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
W20260924 19:57:57.409593 17521 leader_election.cc:336] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.712s (SENT)
I20260924 19:57:57.409971 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
W20260924 19:57:57.416317 17601 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 78801b8d0b664217a74b20160ba2d8ac (table default.Table_454683a3d79d4988b41292f7776df549 [id=81b26242949442c0a7db6f55f0f18097], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 78801b8d0b664217a74b20160ba2d8ac consensus state has no leader
I20260924 19:57:57.440685 18223 tablet_bootstrap.cc:654] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:57.444545 19683 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 19:57:57.453613 18220 tablet_bootstrap.cc:492] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:57.462015 18220 ts_tablet_manager.cc:1403] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.000s
I20260924 19:57:57.464514 17818 raft_consensus.cc:2468] T 70fea649583648ca86b4b3673ab0305f P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:57.464485 18220 raft_consensus.cc:359] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.465107 18220 raft_consensus.cc:385] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:57.465366 18220 raft_consensus.cc:740] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:57.465950 18220 consensus_queue.cc:260] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.472738 19641 raft_consensus.cc:493] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:57.473201 19641 raft_consensus.cc:515] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.476441 19641 leader_election.cc:290] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:57.477922 17818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb111f7a189c4f00a41686d647c64e65" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:57.480443 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb111f7a189c4f00a41686d647c64e65" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:57.480851 18220 ts_tablet_manager.cc:1434] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20260924 19:57:57.481887 18220 tablet_bootstrap.cc:492] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:57.481971 17905 leader_election.cc:304] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:57.482607 19642 raft_consensus.cc:2749] T cb111f7a189c4f00a41686d647c64e65 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:57:57.494573 19529 consensus_queue.cc:1048] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:57.500815 19642 consensus_queue.cc:1048] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:57.515172 18223 tablet_bootstrap.cc:492] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:57.516786 18223 ts_tablet_manager.cc:1403] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.125s user 0.005s sys 0.006s
I20260924 19:57:57.524633 19500 consensus_queue.cc:1048] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:57.537693 19500 consensus_queue.cc:1048] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:57.519891 18223 raft_consensus.cc:359] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:57.554116 18223 raft_consensus.cc:385] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:57.556507 18220 tablet_bootstrap.cc:654] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:57.554908 18223 raft_consensus.cc:740] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:57.563606 18223 consensus_queue.cc:260] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:57.585076 18223 ts_tablet_manager.cc:1434] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.066s user 0.006s sys 0.000s
I20260924 19:57:57.585337 19615 consensus_queue.cc:1048] T bea1ebdf647e4ef9a131a0398672b8ab P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:57.588379 18223 tablet_bootstrap.cc:492] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:57:57.646893 18223 tablet_bootstrap.cc:654] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:57.656296 18220 tablet_bootstrap.cc:492] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:57.657567 18220 ts_tablet_manager.cc:1403] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.176s user 0.010s sys 0.001s
W20260924 19:57:57.675928 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 65889.3us
I20260924 19:57:57.688876 18223 tablet_bootstrap.cc:492] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:57:57.689296 18223 ts_tablet_manager.cc:1403] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.101s user 0.013s sys 0.001s
I20260924 19:57:57.689934 18220 raft_consensus.cc:359] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.699370 18223 raft_consensus.cc:359] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.699990 18223 raft_consensus.cc:385] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:57.700250 18223 raft_consensus.cc:740] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:57:57.700814 18223 consensus_queue.cc:260] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.713297 18220 raft_consensus.cc:385] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:57.714562 18220 raft_consensus.cc:740] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:57.716704 18220 consensus_queue.cc:260] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.723953 18220 ts_tablet_manager.cc:1434] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.064s user 0.005s sys 0.000s
I20260924 19:57:57.728569 18220 tablet_bootstrap.cc:492] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:57.741312 19500 consensus_queue.cc:1048] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:57.756987 18220 tablet_bootstrap.cc:654] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:57.709671 18223 ts_tablet_manager.cc:1434] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20260924 19:57:57.782872 19642 raft_consensus.cc:493] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:57.783176 19500 raft_consensus.cc:493] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:57.783351 19642 raft_consensus.cc:515] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.783552 19500 raft_consensus.cc:515] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.785142 19500 leader_election.cc:290] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:57.786250 19642 leader_election.cc:290] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:57.787478 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fe8d2d06dd4f83bef8c52a82693344" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:57.788154 18110 raft_consensus.cc:2468] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:57.787943 19615 consensus_queue.cc:1048] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:57.790011 17905 leader_election.cc:304] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:57.790704 19615 raft_consensus.cc:2804] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:57.791015 19615 raft_consensus.cc:493] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:57.791366 19615 raft_consensus.cc:3060] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.795871 17819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fe8d2d06dd4f83bef8c52a82693344" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:57.796494 17819 raft_consensus.cc:2468] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:57.799120 19615 raft_consensus.cc:515] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.800742 17809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fe8d2d06dd4f83bef8c52a82693344" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:57.801344 17809 raft_consensus.cc:2468] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:57.803949 18112 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fe8d2d06dd4f83bef8c52a82693344" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:57.804512 18112 raft_consensus.cc:3060] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.804566 19615 leader_election.cc:290] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:57.808099 18220 tablet_bootstrap.cc:492] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:57.808557 18220 ts_tablet_manager.cc:1403] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.080s user 0.014s sys 0.002s
I20260924 19:57:57.811039 18220 raft_consensus.cc:359] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.813105 18220 raft_consensus.cc:385] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:57.813349 18220 raft_consensus.cc:740] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:57.814074 18220 consensus_queue.cc:260] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.816847 18220 ts_tablet_manager.cc:1434] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260924 19:57:57.812525 18112 raft_consensus.cc:2468] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:57.817703 18220 tablet_bootstrap.cc:492] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a: Bootstrap starting.
I20260924 19:57:57.818528 19500 consensus_queue.cc:1048] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:57.818688 17905 leader_election.cc:304] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:57.830277 19642 raft_consensus.cc:2804] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:57.830756 19642 raft_consensus.cc:697] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:57.831576 19642 consensus_queue.cc:237] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.844406 17809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fe8d2d06dd4f83bef8c52a82693344" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:57.844981 17809 raft_consensus.cc:3060] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.851133 19615 consensus_queue.cc:1048] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:57.856791 19642 consensus_queue.cc:1048] T 2de3877b94b44c94b6d0fdad93f8aa79 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:57.862234 19375 raft_consensus.cc:493] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:57.862711 19375 raft_consensus.cc:515] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.870321 18220 tablet_bootstrap.cc:654] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a: Neither blocks nor log segments found. Creating new log.
I20260924 19:57:57.883774 19375 leader_election.cc:290] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:57.886790 19529 consensus_queue.cc:1048] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:57.887774 17965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10fe8d2d06dd4f83bef8c52a82693344" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:57.896351 18220 tablet_bootstrap.cc:492] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a: No bootstrap required, opened a new log
I20260924 19:57:57.897468 18220 ts_tablet_manager.cc:1403] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.003s
I20260924 19:57:57.833932 18051 leader_election.cc:304] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
W20260924 19:57:57.899073 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 66785.1us
I20260924 19:57:57.899147 19500 raft_consensus.cc:2764] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 19:57:57.899926 17965 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97bbb17c170d42349d2c9cdb41349cb9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:57.904203 17965 raft_consensus.cc:2468] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:57.908174 18220 raft_consensus.cc:359] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:57.929078 17810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97bbb17c170d42349d2c9cdb41349cb9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:57.929735 17810 raft_consensus.cc:2468] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:57.921082 19644 raft_consensus.cc:493] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:57.973268 18220 raft_consensus.cc:385] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:57:57.973618 18220 raft_consensus.cc:740] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Initialized, Role: FOLLOWER
I20260924 19:57:57.992247 17809 raft_consensus.cc:2468] T 10fe8d2d06dd4f83bef8c52a82693344 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:57.993110 18220 consensus_queue.cc:260] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:58.004309 18050 leader_election.cc:304] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:58.009373 19500 raft_consensus.cc:2804] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:58.009721 19500 raft_consensus.cc:493] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:58.009989 19500 raft_consensus.cc:3060] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:58.017923 19500 raft_consensus.cc:515] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:58.020576 19500 leader_election.cc:290] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:58.026836 17812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97bbb17c170d42349d2c9cdb41349cb9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:58.027637 17812 raft_consensus.cc:3060] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:57.979700 19642 consensus_queue.cc:1048] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20260924 19:57:58.034183 19642 raft_consensus.cc:493] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:58.034638 19642 raft_consensus.cc:515] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:57.979353 19644 raft_consensus.cc:515] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:58.056411 19644 leader_election.cc:290] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
I20260924 19:57:58.060600 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97bbb17c170d42349d2c9cdb41349cb9" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:58.061267 17972 raft_consensus.cc:3060] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:58.067703 17972 raft_consensus.cc:2468] T 97bbb17c170d42349d2c9cdb41349cb9 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
W20260924 19:57:58.100334 17905 leader_election.cc:343] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 1d95d6e48173436c8c59130464413ea1
I20260924 19:57:58.104223 17812 raft_consensus.cc:2468] T 97bbb17c170d42349d2c9cdb41349cb9 P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
W20260924 19:57:58.107688 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 49795.1us
I20260924 19:57:58.109283 18050 leader_election.cc:304] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:58.110746 19504 raft_consensus.cc:2804] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:58.111335 19504 raft_consensus.cc:697] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:58.112614 19504 consensus_queue.cc:237] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:58.122974 17972 raft_consensus.cc:3060] T fbd5915ee98a4eda94479e9330cf8d04 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:58.129319 19642 leader_election.cc:290] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
I20260924 19:57:58.130648 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d25793b65da4315b6f95726374ea46c" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" is_pre_election: true
I20260924 19:57:58.131235 18110 raft_consensus.cc:2468] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:58.136994 19641 consensus_queue.cc:1048] T 18bb8f3d43df4aae8614b6dba9ffc668 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:58.148440 17905 leader_election.cc:304] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:58.149201 19715 raft_consensus.cc:2804] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:58.149571 19715 raft_consensus.cc:493] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:58.149946 19715 raft_consensus.cc:3060] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:58.164465 19715 raft_consensus.cc:515] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:58.167522 18110 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d25793b65da4315b6f95726374ea46c" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35"
I20260924 19:57:58.168083 18110 raft_consensus.cc:3060] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:58.172047 18220 ts_tablet_manager.cc:1434] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a: Time spent starting tablet: real 0.273s user 0.004s sys 0.007s
I20260924 19:57:58.176111 19715 leader_election.cc:290] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Requested vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:57:58.178464 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 51388.4us
W20260924 19:57:58.179208 17906 leader_election.cc:343] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 1d95d6e48173436c8c59130464413ea1
I20260924 19:57:58.179704 17906 leader_election.cc:304] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:57:58.181151 19615 raft_consensus.cc:2749] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:57:58.187336 19500 consensus_queue.cc:1048] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:58.187598 19718 consensus_queue.cc:1048] T 144355b18c2f413baa28dd632623db52 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:58.206696 18110 raft_consensus.cc:2468] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
I20260924 19:57:58.220584 17905 leader_election.cc:304] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:58.221725 19715 raft_consensus.cc:2804] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:58.224349 17809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d25793b65da4315b6f95726374ea46c" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:58.225071 17809 raft_consensus.cc:2468] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 0.
I20260924 19:57:58.226132 17812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d25793b65da4315b6f95726374ea46c" candidate_uuid: "f74e6ed351284d7fb110e44336ce643a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:58.226653 17812 raft_consensus.cc:3060] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:58.223222 19715 raft_consensus.cc:697] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Becoming Leader. State: Replica: f74e6ed351284d7fb110e44336ce643a, State: Running, Role: LEADER
I20260924 19:57:58.240927 19715 consensus_queue.cc:237] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:57:58.262449 19504 raft_consensus.cc:493] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:57:58.263100 19504 raft_consensus.cc:515] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:58.265192 19504 leader_election.cc:290] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:58.275393 17812 raft_consensus.cc:2468] T 3d25793b65da4315b6f95726374ea46c P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74e6ed351284d7fb110e44336ce643a in term 1.
W20260924 19:57:58.276836 18903 leader_election.cc:336] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.549s (SENT)
I20260924 19:57:58.277323 18903 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:57:58.279469 17963 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00e3626ea154dc5a959f0fbb4e849dc" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a" is_pre_election: true
I20260924 19:57:58.280175 17963 raft_consensus.cc:2468] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
W20260924 19:57:58.287842 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 110074us
I20260924 19:57:58.298702 18050 leader_election.cc:304] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaf64bfb64aa42948c9bc1411ce4ce35, f74e6ed351284d7fb110e44336ce643a; no voters:
I20260924 19:57:58.300407 19672 raft_consensus.cc:2804] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:57:58.300773 19672 raft_consensus.cc:493] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:57:58.301076 19672 raft_consensus.cc:3060] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:58.303398 19504 consensus_queue.cc:1048] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:58.313385 19672 raft_consensus.cc:515] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:58.315960 19672 leader_election.cc:290] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Requested vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301)
I20260924 19:57:58.322250 17809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00e3626ea154dc5a959f0fbb4e849dc" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202" is_pre_election: true
I20260924 19:57:58.323101 17809 raft_consensus.cc:2468] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 0.
I20260924 19:57:58.338632 17809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00e3626ea154dc5a959f0fbb4e849dc" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f6e3a7b06148918d230c77ab0c8202"
I20260924 19:57:58.341856 19731 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 19:57:58.343683 19500 consensus_queue.cc:1048] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:58.351171 18903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.188:38761, user_credentials={real_user=slave}} blocked reactor thread for 74365.5us
I20260924 19:57:58.339250 17809 raft_consensus.cc:3060] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:58.368983 17809 raft_consensus.cc:2468] T a00e3626ea154dc5a959f0fbb4e849dc P c4f6e3a7b06148918d230c77ab0c8202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:58.397828 18051 leader_election.cc:304] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35; no voters:
I20260924 19:57:58.398880 19672 raft_consensus.cc:2804] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:57:58.399678 19672 raft_consensus.cc:697] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Becoming Leader. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Running, Role: LEADER
I20260924 19:57:58.400014 17972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a00e3626ea154dc5a959f0fbb4e849dc" candidate_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74e6ed351284d7fb110e44336ce643a"
I20260924 19:57:58.400597 17972 raft_consensus.cc:3060] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:57:58.400446 19672 consensus_queue.cc:237] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:57:58.421912 17972 raft_consensus.cc:2468] T a00e3626ea154dc5a959f0fbb4e849dc P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaf64bfb64aa42948c9bc1411ce4ce35 in term 1.
I20260924 19:57:58.430866 19375 consensus_queue.cc:1048] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:58.490561 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 49548.2us
I20260924 19:57:58.495333 19672 consensus_queue.cc:1048] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:58.702171 19715 consensus_queue.cc:1048] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:58.720731 19642 consensus_queue.cc:1048] T 10fe8d2d06dd4f83bef8c52a82693344 P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:58.997431 19504 consensus_queue.cc:1048] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:59.059105 19718 consensus_queue.cc:1048] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:59.219964 19748 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:57:59.220572 19748 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:57:59.223610 19642 consensus_queue.cc:1048] T 3d25793b65da4315b6f95726374ea46c P f74e6ed351284d7fb110e44336ce643a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:57:59.224298 18939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:57:59.294740 19748 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:57:59.343895 19672 consensus_queue.cc:1048] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:57:59.548563 19500 consensus_queue.cc:1048] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:59.677673 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 48889.3us
I20260924 19:57:59.704679 19375 consensus_queue.cc:1048] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:57:59.885349 19756 log.cc:927] Time spent T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260924 19:58:00.009135 17521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.712s (SENT) [suppressed 1 similar messages]
W20260924 19:58:00.008632 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 74481.5us
W20260924 19:58:00.075428 17521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.188:38761, user_credentials={real_user=slave}} blocked reactor thread for 66306.4us
W20260924 19:58:00.139992 17906 consensus_peers.cc:597] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae7a0660795e4e4b87d0959ca84d5d1b. This is attempt 6: this message will repeat every 5th retry.
W20260924 19:58:00.659399 18051 consensus_peers.cc:597] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57008266b80345e0a3f8e60bd7e2d62b. This is attempt 6: this message will repeat every 5th retry.
W20260924 19:58:00.789762 17521 leader_election.cc:336] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.712s (SENT)
I20260924 19:58:00.790367 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:58:00.831766 19765 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 19:58:00.924960 19766 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:58:00.925729 19766 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:58:00.933698 19766 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:58:00.937856 17536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:58:01.163468 19642 raft_consensus.cc:493] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:01.164356 19642 raft_consensus.cc:515] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:58:01.167100 19642 leader_election.cc:290] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:58:01.170740 17906 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
W20260924 19:58:01.191490 17905 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:58:01.191958 17905 leader_election.cc:304] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:58:01.193938 19767 raft_consensus.cc:2749] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:58:01.386395 17905 consensus_peers.cc:597] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a -> Peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Couldn't send request to peer eaf64bfb64aa42948c9bc1411ce4ce35. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae7a0660795e4e4b87d0959ca84d5d1b. This is attempt 6: this message will repeat every 5th retry.
W20260924 19:58:01.523741 18903 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.531s (SENT) [suppressed 1 similar messages]
I20260924 19:58:01.612874 19642 raft_consensus.cc:493] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:01.613425 19642 raft_consensus.cc:515] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:58:01.615731 19642 leader_election.cc:290] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:58:01.618263 17905 leader_election.cc:343] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 44199d2655f14dcf97afe2180789aae4
W20260924 19:58:01.632454 17906 leader_election.cc:343] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 44199d2655f14dcf97afe2180789aae4
I20260924 19:58:01.632903 17906 leader_election.cc:304] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:58:01.634019 19642 raft_consensus.cc:2749] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:58:01.680183 17906 consensus_peers.cc:597] T 3192f66a492442c1bf4137cfe7e13bc9 P f74e6ed351284d7fb110e44336ce643a -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3192f66a492442c1bf4137cfe7e13bc9. This is attempt 6: this message will repeat every 5th retry.
W20260924 19:58:02.095450 18903 leader_election.cc:336] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.531s (SENT)
I20260924 19:58:02.096009 18903 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:58:02.130842 19778 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 19:58:02.146349 18903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.188:38761, user_credentials={real_user=slave}} blocked reactor thread for 46563.4us
I20260924 19:58:02.574734 19782 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:58:02.577277 19782 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:58:02.581692 18939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:58:02.588174 19782 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:58:03.531426 19795 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:58:03.532018 19795 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:58:03.534422 19795 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:58:03.536177 17536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
W20260924 19:58:03.659394 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 46131.9us
W20260924 19:58:03.848229 17521 leader_election.cc:336] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.603s (SENT)
I20260924 19:58:03.848724 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:58:03.876551 19796 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 19:58:04.175858 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 47122.7us
W20260924 19:58:04.224802 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 52080.9us
I20260924 19:58:04.272114 19798 raft_consensus.cc:493] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:04.272607 19798 raft_consensus.cc:515] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:58:04.274406 19798 leader_election.cc:290] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:58:04.278648 17906 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 8aef0359a77240679240823612507ee8
W20260924 19:58:04.287180 17905 leader_election.cc:343] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 8aef0359a77240679240823612507ee8
I20260924 19:58:04.287601 17905 leader_election.cc:304] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:58:04.293691 19786 raft_consensus.cc:2749] T 8aef0359a77240679240823612507ee8 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:04.597085 19801 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:58:04.601747 19801 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:58:04.609992 18939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:58:04.619295 19801 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
W20260924 19:58:04.644501 18903 leader_election.cc:336] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.539s (SENT)
I20260924 19:58:04.645051 18903 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:58:04.662743 19802 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 19:58:05.069197 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 140208us
W20260924 19:58:05.307928 17521 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.688s (SENT) [suppressed 1 similar messages]
I20260924 19:58:05.691257 19810 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:58:05.695437 19810 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:58:05.700070 17536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:58:05.701499 19810 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:58:05.919320 19805 raft_consensus.cc:493] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 19:58:05.919901 17905 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 57723.6us
I20260924 19:58:05.919811 19805 raft_consensus.cc:515] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:58:05.921782 19805 leader_election.cc:290] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:58:05.927239 17906 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
W20260924 19:58:05.924862 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 58860.3us
W20260924 19:58:05.958014 17905 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:58:05.966549 17905 leader_election.cc:304] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:58:05.969821 19805 raft_consensus.cc:2749] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:58:06.040056 17521 leader_election.cc:336] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.688s (SENT)
I20260924 19:58:06.040506 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:58:06.059077 19812 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 19:58:06.373157 18903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.188:38761, user_credentials={real_user=slave}} blocked reactor thread for 43673.8us
I20260924 19:58:06.655895 19804 raft_consensus.cc:493] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:06.656373 19804 raft_consensus.cc:515] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:58:06.660234 19804 leader_election.cc:290] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465), eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575)
W20260924 19:58:06.662853 17906 leader_election.cc:343] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: 1d95d6e48173436c8c59130464413ea1
W20260924 19:58:06.667104 17905 leader_election.cc:343] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: 1d95d6e48173436c8c59130464413ea1
I20260924 19:58:06.669111 17905 leader_election.cc:304] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:58:06.679054 19805 raft_consensus.cc:2749] T 1d95d6e48173436c8c59130464413ea1 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:58:06.826123 18051 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 77585.2us
I20260924 19:58:06.863894 19819 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:58:06.864506 19819 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:58:06.899581 19819 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:58:06.900370 18939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
W20260924 19:58:07.004916 18903 leader_election.cc:336] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.630s (SENT)
I20260924 19:58:07.005367 18903 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
W20260924 19:58:07.082473 18903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.188:38761, user_credentials={real_user=slave}} blocked reactor thread for 76885.1us
I20260924 19:58:07.125133 19820 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 19:58:07.335316 17599 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet f169c47d711d4566b2004cdfd0de725d consensus state has no leader
W20260924 19:58:07.379480 17753 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:40575, user_credentials={real_user=slave}} blocked reactor thread for 226107us
I20260924 19:58:07.947624 19830 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:58:07.948248 19830 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:58:07.951138 17536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:58:07.983934 19830 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
W20260924 19:58:08.387140 17521 leader_election.cc:336] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.720s (SENT)
I20260924 19:58:08.387598 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:58:08.426893 19834 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260924 19:58:08.670832 18903 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.684s (SENT) [suppressed 2 similar messages]
I20260924 19:58:09.146235 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_65d20bb187ea4069b2e589e7643028be [id=00e4a8cdcb374ae6a4446b3b7404d7a8]
I20260924 19:58:09.148093 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_5cc282e71cd74150a849aff1620d54f1 [id=010f7a5a46034f008e83a6492524b778]
I20260924 19:58:09.154577 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_e91b551877474bb0914b854a58447b3a [id=0606053a508145868b640f364fbf48d6]
I20260924 19:58:09.159615 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_db4a5e39fd954bb0b4c271899feefe90 [id=0b7e96c71a074f838a519746d9c92c3f]
I20260924 19:58:09.171883 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_081e10d31dc74797abba37099d8a75fc [id=0ea22c5631c347aa9c7aa0e24140cb14]
I20260924 19:58:09.203289 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_05427967e4cc480cba4d8f7128b23677 [id=11bd6526602f4c0dbdb90bbc8806262b]
I20260924 19:58:09.224277 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_6b68af48dd4140dda7d1854a3cf2269f [id=1a5ed525f4364782a0793571d4caa5cb]
I20260924 19:58:09.227317 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_39ac82df7ce14073912622fcd1b57aaa [id=1ae05845cc1641ffaf3b82b0f1390e43]
I20260924 19:58:09.270010 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_202c25f88e1a485c9ca6677d9cf53e75 [id=221fc6b9725747ada3ed73dc569052cb]
I20260924 19:58:09.278132 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_fcbe3b81791a4aaf923847b1366eb6f9 [id=25cb87a38f0f473d99bad6f06d4edcf4]
I20260924 19:58:09.285941 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_66edcc97f7a64dfa85bf24d26e49c5de [id=31d1abb2c65744d78c5e61dd4ae81b63]
I20260924 19:58:09.295401 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]
I20260924 19:58:09.297595 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b]
I20260924 19:58:09.299772 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265]
I20260924 19:58:09.301057 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_9faf28b24a2c47449a862536ea873f36 [id=3d02208bd3604c729b474a4f005b8cf8]
I20260924 19:58:09.302495 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]
I20260924 19:58:09.306046 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_1dad27c8fd4445fb89e32722eef0abe2 [id=4cd61b434f884eed908abf700d02cdd0]
I20260924 19:58:09.313962 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_b2824cbbb81c4186840b758c61eccbb2 [id=4e169b185baf4e7a81735a13b6d82fca]
I20260924 19:58:09.315441 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_e3a6abe9c34a4fd6a8567a9c0a8190c3 [id=4e80d69486a34b848393f01e656d122b]
I20260924 19:58:09.316817 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_9259880c400d44d69831fe68d31b06f4 [id=4f8c78c1109f4245b3ff41258543de1e]
I20260924 19:58:09.318185 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_9b8f6ad547fc4c3fa379c93590aebd56 [id=4fc8e565a6a640549716cee596fa3f9d]
I20260924 19:58:09.335582 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]
I20260924 19:58:09.337119 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_5e789131162643caa4c014a55244066f [id=52ff52bfd3904ab0b4826673127f70ac]
I20260924 19:58:09.338495 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_5e50f3dbbb674d658a5afac0ebcedaa6 [id=545488f05d324b748e42dae4a665f0d6]
I20260924 19:58:09.340519 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140]
I20260924 19:58:09.341861 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_93373f379bb14b6ca7a3ac7a628cad63 [id=5878a9a1b1d54055a5e8dc712b9de665]
I20260924 19:58:09.345564 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_5aeda90fd8e3425a8c49a239533690c9 [id=63622937f5a64a05b56aa1310407ec99]
I20260924 19:58:09.350315 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_7c27cdce0a3b40f69142de52aaf39493 [id=65f397bcbc164d0caa6cecae1bab74da]
I20260924 19:58:09.353160 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_ca51e47e4fc44d9ba5dd07cbeefa6e68 [id=6ea0f9084fd84e848fdecdd696cdc773]
I20260924 19:58:09.354445 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_388c160758d44e1e89b8cb5d541316f8 [id=70f30a75b8d443a0ae28243adf40862a]
I20260924 19:58:09.356465 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_88b623b6a00845afb22e24e34604e9ea [id=72c7d8cc98ee49679d8c7589a9ac0c8f]
I20260924 19:58:09.357777 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f]
I20260924 19:58:09.362594 19841 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
I20260924 19:58:09.363134 19841 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:58:09.376343 19841 leader_election.cc:290] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 3e059bbe4dac4178bccb3be48cb1118d (127.17.2.190:46309), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
W20260924 19:58:09.376497 18903 leader_election.cc:336] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761): Timed out: RequestConsensusVote RPC to 127.17.2.188:38761 timed out after 0.684s (SENT)
I20260924 19:58:09.376931 18903 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:58:09.377856 18939 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "3e059bbe4dac4178bccb3be48cb1118d" is_pre_election: true
I20260924 19:58:09.359014 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_a82c256e36e448c296c86a30176f4135 [id=757ccfb2d39f4907a2580b365d283ace]
I20260924 19:58:09.384810 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_0c6b7fd4e7bd42a3ba0d407aea438273 [id=79e6d48b52b740958363f06cf05f2f63]
I20260924 19:58:09.386200 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_a10031a0fa004ac3ad48e6020477494b [id=7aab4014a5bc4823bbe845c021153fc9]
I20260924 19:58:09.392103 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b [id=7babe5418e104e76871835ca3e30f95c]
I20260924 19:58:09.393451 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_5208198cb618443dbec0f760012e4edd [id=7cb855f3577d4824ac7edb72b8f7072c]
I20260924 19:58:09.394740 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_aeac9df2b26b490a83f10d672335d75e [id=80222cdc26da492e9f386e9eb2fb77b8]
I20260924 19:58:09.396239 19842 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 19:58:09.400348 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_454683a3d79d4988b41292f7776df549 [id=81b26242949442c0a7db6f55f0f18097]
I20260924 19:58:09.401648 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_ae1050d7716043fbad49a8f62505e719 [id=8681c352f02242a69511794ee585bf28]
I20260924 19:58:09.409941 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_e3179b11ea114ce99779ed0b2ac46732 [id=8c39c8f34038451d840db3c909d76f3d]
I20260924 19:58:09.411276 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_2a8be69ab9604081a07df5197e7fab9d [id=8d382cdb38764b11ba323546fff603bf]
I20260924 19:58:09.412633 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]
I20260924 19:58:09.422385 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_7b49c63ad79c4132afbbbb9e1f2d4ca0 [id=8edea4510fd84d40819b48b401ccfe64]
I20260924 19:58:09.425794 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_a12f8684de2041ec83f2fee77829afdd [id=94cea8de30ca4010a95f142ac8408653]
I20260924 19:58:09.428632 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_b6d3fa192eb741158adf5b69323a29de [id=9ead6eed50694fd1b920b5b0d46891c9]
I20260924 19:58:09.429955 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_0a22025767fb456485c2624d5ca0dcbb [id=9f151548ce074cb6b8b8497fa246d61a]
I20260924 19:58:09.432691 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_c42c4d9b6c1e4d2abd6aa51f251be773 [id=a72d39e753b2435a800e68d308cdc86c]
I20260924 19:58:09.434707 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06]
I20260924 19:58:09.450664 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]
I20260924 19:58:09.452812 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_4e897947eb9e4466a9f050091b6a7b60 [id=b7032f43d384487d9b041986780669c0]
I20260924 19:58:09.454087 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_954dba173a70455b90daee9230b44b6f [id=b743c3e3916a45069fabeed784e9e249]
I20260924 19:58:09.455415 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_14bf9267f2da4ba48354027fc92cab3d [id=b898f46c59b246e39fe8b62a45019ecb]
I20260924 19:58:09.456687 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_415dca00ed274e59961ddf45e42de178 [id=b8a8b30617164fbcb9f6918142768362]
I20260924 19:58:09.457954 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_b8cbd748636249c0ab17657b8f259893 [id=b9bee2849a334d098dd5d52fed7f6cc0]
I20260924 19:58:09.489353 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_a7d63063e7ee4493abe8c76da6d4f99d [id=be19c62729a34ae1a15966662303a869]
I20260924 19:58:09.490819 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_11ca95d108694c9498d4de67c9ff8808 [id=bec51c14454046329b3eb7016389f235]
I20260924 19:58:09.492161 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_a50ccb6bd6a042bc99b503e5aa1188ad [id=bfd5e6e55fb049f9965b190374b38c3d]
I20260924 19:58:09.514679 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_843b88f433f044458daa98228935a46e [id=c19508f168ed4191bdd25de1107ab416]
I20260924 19:58:09.516830 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_1d9dfc2ce03046ba955d3d82049402fd [id=c44dfe41598f4e6cbf320e3044b49fe5]
I20260924 19:58:09.518028 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_954c770a550c49599e38ee3850e4aea1 [id=c4ad22b552204e2d8209b275e57eb9c4]
I20260924 19:58:09.519197 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_bdae18e3011a421d96cdcc107e8f1dc2 [id=ca7affed79d94a2597fdaa1ff38e5e32]
I20260924 19:58:09.520349 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_fbc5ceeb4d1242d19344825cefb6ef2a [id=caf8f776b4754b80a6aac8211f1b7a6f]
I20260924 19:58:09.537264 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_8be3db507b2d42da90969bacc361c127 [id=cd5ff543cb1f48f58d5867aef9706963]
I20260924 19:58:09.539275 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df]
I20260924 19:58:09.540454 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_bb9775fce761443ca3a306e0cace741e [id=dc2b2e2afb22446ba04263f4f86a23e0]
I20260924 19:58:09.542243 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_dea06619a5364df0961e47cb617c9e52 [id=e40a55e03a2d4ceea51d555d0c60a334]
I20260924 19:58:09.550902 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_83714385573a44b2af65ac658fbe2cfa [id=e5b211f0af7143709eec620ecf04f2b5]
I20260924 19:58:09.552122 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_5128840101ca460b9e13dfd054764e9d [id=e8a1bc657d9b4fb2b1f0eb458d41dbbd]
I20260924 19:58:09.553880 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_0673b53c69bf4c429697ba739ef113e4 [id=ea252b103a0f43d18971c78fe2174c2e]
I20260924 19:58:09.555016 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_3bf03540f35b468dacc7a86dd2d3d2a7 [id=eaf422968e464f1986f89b6944a6adbf]
I20260924 19:58:09.579849 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_8ade2ea47f344fd18a687f7d5e0ea568 [id=eecec62f4e6a4e56920d841de1b9077b]
I20260924 19:58:09.582401 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_2bb6cac7492d4c5ca461d126788c9722 [id=f1f88aa92d884988b4bdf211754de8de]
I20260924 19:58:09.595211 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe]
I20260924 19:58:09.602129 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_771765c449ce492986addb028e1b11d1 [id=f954ad85289641fda971e5c5155b9147]
I20260924 19:58:09.603405 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_b7e4d07630f24ee0ab58d5269687448d [id=f9f9606f84804c769a867198983fb713]
I20260924 19:58:09.604574 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_4e3a3158cde24302806b7ba7827831c4 [id=fa0b10df171340868b0305a28d192ea5]
I20260924 19:58:09.605739 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_88352246addd43f4be871c983413b2de [id=fa469a44fb3741d39036337129cc36a1]
I20260924 19:58:09.606890 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_2255ca45bfcf4f819376770df3186e24 [id=fb5bd2d87d78475d803e753473de0c7a]
I20260924 19:58:09.626778 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_ee93a1803fd54a4ba6723d1e3783c004 [id=fec7f8aa891e42c1a84fe32fe4e0c44f]
I20260924 19:58:09.628155 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_c449d7c2b02a4bcdb7091a3560bcaa02 [id=ff6c3885d44548b78bb3e4c556a5f383]
I20260924 19:58:09.647261 19467 catalog_manager.cc:692] Loaded metadata for table default.Table_a36de9161f8f4ea08243a7222c099822 [id=ff9a4ba6b88648748c228135ed440066]
I20260924 19:58:09.715663 19467 tablet_loader.cc:96] loaded metadata for tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82])
I20260924 19:58:09.718808 19467 tablet_loader.cc:96] loaded metadata for tablet 0b99ff1920274831b58d8283bd592818 (table default.Table_aeac9df2b26b490a83f10d672335d75e [id=80222cdc26da492e9f386e9eb2fb77b8])
I20260924 19:58:09.719955 19467 tablet_loader.cc:96] loaded metadata for tablet 0c08f10d2a364cb8a0a7ea40654c3fe4 (table default.Table_2bb6cac7492d4c5ca461d126788c9722 [id=f1f88aa92d884988b4bdf211754de8de])
I20260924 19:58:09.721060 19467 tablet_loader.cc:96] loaded metadata for tablet 0c6943a365da464e83cedc2b9711ca89 (table default.Table_93373f379bb14b6ca7a3ac7a628cad63 [id=5878a9a1b1d54055a5e8dc712b9de665])
I20260924 19:58:09.722133 19467 tablet_loader.cc:96] loaded metadata for tablet 0cf2df461598411f8ad9f3a51eafc6f7 (table default.Table_a10031a0fa004ac3ad48e6020477494b [id=7aab4014a5bc4823bbe845c021153fc9])
I20260924 19:58:09.755924 19467 tablet_loader.cc:96] loaded metadata for tablet 10fe8d2d06dd4f83bef8c52a82693344 (table default.Table_fcbe3b81791a4aaf923847b1366eb6f9 [id=25cb87a38f0f473d99bad6f06d4edcf4])
I20260924 19:58:09.757151 19467 tablet_loader.cc:96] loaded metadata for tablet 144355b18c2f413baa28dd632623db52 (table default.Table_e01b276a7ed44a0b966d8f5c2b7ac53b [id=7babe5418e104e76871835ca3e30f95c])
I20260924 19:58:09.758955 19467 tablet_loader.cc:96] loaded metadata for tablet 18bb8f3d43df4aae8614b6dba9ffc668 (table default.Table_2255ca45bfcf4f819376770df3186e24 [id=fb5bd2d87d78475d803e753473de0c7a])
I20260924 19:58:09.792855 19467 tablet_loader.cc:96] loaded metadata for tablet 274f3decdc7e42c0917bf28151c4da71 (table default.Table_954c770a550c49599e38ee3850e4aea1 [id=c4ad22b552204e2d8209b275e57eb9c4])
I20260924 19:58:09.793917 19467 tablet_loader.cc:96] loaded metadata for tablet 28fb29faeeb34c57bb6e52a9c5ffd218 (table default.Table_843b88f433f044458daa98228935a46e [id=c19508f168ed4191bdd25de1107ab416])
I20260924 19:58:09.802115 19467 tablet_loader.cc:96] loaded metadata for tablet 2a1ac7e1b192465696640c02898bc39f (table default.Table_899397c3fffc4ae4b4d59756c4ab73cb [id=37803adb9d6349619ebbc92154ca200b])
I20260924 19:58:09.803189 19467 tablet_loader.cc:96] loaded metadata for tablet 2b0b011357674a42a09d83f9384abaf1 (table default.Table_5aeda90fd8e3425a8c49a239533690c9 [id=63622937f5a64a05b56aa1310407ec99])
I20260924 19:58:09.804960 19467 tablet_loader.cc:96] loaded metadata for tablet 2de3877b94b44c94b6d0fdad93f8aa79 (table default.Table_65d20bb187ea4069b2e589e7643028be [id=00e4a8cdcb374ae6a4446b3b7404d7a8])
I20260924 19:58:09.806051 19467 tablet_loader.cc:96] loaded metadata for tablet 2e6d926ff7c24901a15b0c7b2239fbad (table default.Table_7c27cdce0a3b40f69142de52aaf39493 [id=65f397bcbc164d0caa6cecae1bab74da])
I20260924 19:58:09.815529 19467 tablet_loader.cc:96] loaded metadata for tablet 335f979eb04c49bcb53023693f54ad41 (table default.Table_ae1050d7716043fbad49a8f62505e719 [id=8681c352f02242a69511794ee585bf28])
I20260924 19:58:09.817885 19467 tablet_loader.cc:96] loaded metadata for tablet 3b328d37d55e4116a29322caaa21d431 (table default.Table_4e897947eb9e4466a9f050091b6a7b60 [id=b7032f43d384487d9b041986780669c0])
I20260924 19:58:09.819636 19467 tablet_loader.cc:96] loaded metadata for tablet 3d25793b65da4315b6f95726374ea46c (table default.Table_5cc282e71cd74150a849aff1620d54f1 [id=010f7a5a46034f008e83a6492524b778])
I20260924 19:58:09.820586 19467 tablet_loader.cc:96] loaded metadata for tablet 3e52ecf08c6c45d4b8ba423358fb5fc1 (table default.Table_dea06619a5364df0961e47cb617c9e52 [id=e40a55e03a2d4ceea51d555d0c60a334])
I20260924 19:58:09.821489 19467 tablet_loader.cc:96] loaded metadata for tablet 3eb3d893415b4dfcbcacd435c3c16e5e (table default.Table_c42c4d9b6c1e4d2abd6aa51f251be773 [id=a72d39e753b2435a800e68d308cdc86c])
I20260924 19:58:09.843375 19467 tablet_loader.cc:96] loaded metadata for tablet 431d298f513d47268284d50af2ea1436 (table default.Table_6b68af48dd4140dda7d1854a3cf2269f [id=1a5ed525f4364782a0793571d4caa5cb])
I20260924 19:58:09.845741 19467 tablet_loader.cc:96] loaded metadata for tablet 46e703a4892f445bb0a1c9f5a46db63d (table default.Table_ee93a1803fd54a4ba6723d1e3783c004 [id=fec7f8aa891e42c1a84fe32fe4e0c44f])
I20260924 19:58:09.846837 19467 tablet_loader.cc:96] loaded metadata for tablet 484517ee7df8463bb70d201ead1c6835 (table default.Table_771765c449ce492986addb028e1b11d1 [id=f954ad85289641fda971e5c5155b9147])
I20260924 19:58:09.847796 19467 tablet_loader.cc:96] loaded metadata for tablet 492eb083f50e40a388d2174d6dc8445b (table default.Table_9faf28b24a2c47449a862536ea873f36 [id=3d02208bd3604c729b474a4f005b8cf8])
I20260924 19:58:09.848735 19467 tablet_loader.cc:96] loaded metadata for tablet 56295c0109ce489fb03d65674df62c44 (table default.Table_b7e4d07630f24ee0ab58d5269687448d [id=f9f9606f84804c769a867198983fb713])
I20260924 19:58:09.850534 19467 tablet_loader.cc:96] loaded metadata for tablet 57102b1727e94a90981509c82c082441 (table default.Table_2a8be69ab9604081a07df5197e7fab9d [id=8d382cdb38764b11ba323546fff603bf])
I20260924 19:58:09.851608 19467 tablet_loader.cc:96] loaded metadata for tablet 5df3dcf4ddcf4efa9aa047da56dfb7a7 (table default.Table_a82c256e36e448c296c86a30176f4135 [id=757ccfb2d39f4907a2580b365d283ace])
I20260924 19:58:09.852661 19467 tablet_loader.cc:96] loaded metadata for tablet 5e9f9b76a58e47c0beee5ce94a44b707 (table default.Table_c449d7c2b02a4bcdb7091a3560bcaa02 [id=ff6c3885d44548b78bb3e4c556a5f383])
I20260924 19:58:09.854980 19467 tablet_loader.cc:96] loaded metadata for tablet 66599c38b1a64ad28fe7bd8e5b5056a8 (table default.Table_a50ccb6bd6a042bc99b503e5aa1188ad [id=bfd5e6e55fb049f9965b190374b38c3d])
I20260924 19:58:09.856026 19467 tablet_loader.cc:96] loaded metadata for tablet 6b7f2dd8ba2a4cb190e9ff92a542bacb (table default.Table_11ca95d108694c9498d4de67c9ff8808 [id=bec51c14454046329b3eb7016389f235])
I20260924 19:58:09.857050 19467 tablet_loader.cc:96] loaded metadata for tablet 6bd442f16492495f84bb8363b9bade97 (table default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f])
I20260924 19:58:09.858788 19467 tablet_loader.cc:96] loaded metadata for tablet 6fd0d25961d74b67a42323a045f06b61 (table default.Table_fbc5ceeb4d1242d19344825cefb6ef2a [id=caf8f776b4754b80a6aac8211f1b7a6f])
I20260924 19:58:09.872905 19467 tablet_loader.cc:96] loaded metadata for tablet 70fea649583648ca86b4b3673ab0305f (table default.Table_66edcc97f7a64dfa85bf24d26e49c5de [id=31d1abb2c65744d78c5e61dd4ae81b63])
I20260924 19:58:09.874663 19467 tablet_loader.cc:96] loaded metadata for tablet 7635c52883ec4434bad0cc15f8733f72 (table default.Table_0673b53c69bf4c429697ba739ef113e4 [id=ea252b103a0f43d18971c78fe2174c2e])
I20260924 19:58:09.875782 19467 tablet_loader.cc:96] loaded metadata for tablet 78801b8d0b664217a74b20160ba2d8ac (table default.Table_454683a3d79d4988b41292f7776df549 [id=81b26242949442c0a7db6f55f0f18097])
I20260924 19:58:09.876788 19467 tablet_loader.cc:96] loaded metadata for tablet 7ba365c977984e68a316356465b1e36a (table default.Table_05427967e4cc480cba4d8f7128b23677 [id=11bd6526602f4c0dbdb90bbc8806262b])
I20260924 19:58:09.877843 19467 tablet_loader.cc:96] loaded metadata for tablet 7c1afbfc7de7480ab4be40680a3e3ada (table default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027])
I20260924 19:58:09.878887 19467 tablet_loader.cc:96] loaded metadata for tablet 7c7b8422753a4746bde71fcb25409949 (table default.Table_388c160758d44e1e89b8cb5d541316f8 [id=70f30a75b8d443a0ae28243adf40862a])
I20260924 19:58:09.886005 19467 tablet_loader.cc:96] loaded metadata for tablet 83cfc9998eb84e66b61e88b010992e3e (table default.Table_0c6b7fd4e7bd42a3ba0d407aea438273 [id=79e6d48b52b740958363f06cf05f2f63])
I20260924 19:58:09.887775 19467 tablet_loader.cc:96] loaded metadata for tablet 8523125e35554830aff04b17677d5ceb (table default.Table_5e789131162643caa4c014a55244066f [id=52ff52bfd3904ab0b4826673127f70ac])
I20260924 19:58:09.902184 19467 tablet_loader.cc:96] loaded metadata for tablet 872bc32b9e384c828a0699079b27530d (table default.Table_3bf03540f35b468dacc7a86dd2d3d2a7 [id=eaf422968e464f1986f89b6944a6adbf])
I20260924 19:58:09.903393 19467 tablet_loader.cc:96] loaded metadata for tablet 89646893588340038f30ed4754623ec2 (table default.Table_bb9775fce761443ca3a306e0cace741e [id=dc2b2e2afb22446ba04263f4f86a23e0])
I20260924 19:58:09.906608 19467 tablet_loader.cc:96] loaded metadata for tablet 8c05ec2ba01448d998c6ffa1525de984 (table default.Table_14bf9267f2da4ba48354027fc92cab3d [id=b898f46c59b246e39fe8b62a45019ecb])
I20260924 19:58:09.912828 19467 tablet_loader.cc:96] loaded metadata for tablet 8ea2385391d34f8abd9d105fbd7f22cf (table default.Table_a36de9161f8f4ea08243a7222c099822 [id=ff9a4ba6b88648748c228135ed440066])
I20260924 19:58:09.925766 19467 tablet_loader.cc:96] loaded metadata for tablet 924bc897f48b44069f1fb2ee71e26db7 (table default.Table_415dca00ed274e59961ddf45e42de178 [id=b8a8b30617164fbcb9f6918142768362])
I20260924 19:58:09.926970 19467 tablet_loader.cc:96] loaded metadata for tablet 92c0daa7d67e4aa390e334c669adda76 (table default.Table_88b623b6a00845afb22e24e34604e9ea [id=72c7d8cc98ee49679d8c7589a9ac0c8f])
I20260924 19:58:09.928076 19467 tablet_loader.cc:96] loaded metadata for tablet 954178b8cc1e434085337f66b0e85672 (table default.Table_7b49c63ad79c4132afbbbb9e1f2d4ca0 [id=8edea4510fd84d40819b48b401ccfe64])
I20260924 19:58:09.929152 19467 tablet_loader.cc:96] loaded metadata for tablet 96379e7356ab40218c6ee2c126c0b4ae (table default.Table_39ac82df7ce14073912622fcd1b57aaa [id=1ae05845cc1641ffaf3b82b0f1390e43])
I20260924 19:58:09.931217 19467 tablet_loader.cc:96] loaded metadata for tablet 97bbb17c170d42349d2c9cdb41349cb9 (table default.Table_b2824cbbb81c4186840b758c61eccbb2 [id=4e169b185baf4e7a81735a13b6d82fca])
I20260924 19:58:09.932154 19467 tablet_loader.cc:96] loaded metadata for tablet 9936ffa261ea4aa98426b00a4ef3c6a9 (table default.Table_b6d3fa192eb741158adf5b69323a29de [id=9ead6eed50694fd1b920b5b0d46891c9])
I20260924 19:58:09.935285 19467 tablet_loader.cc:96] loaded metadata for tablet a00e3626ea154dc5a959f0fbb4e849dc (table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3])
I20260924 19:58:09.949802 19467 tablet_loader.cc:96] loaded metadata for tablet a9e7c5b28fea4dbda802702c8b85ec54 (table default.Table_5208198cb618443dbec0f760012e4edd [id=7cb855f3577d4824ac7edb72b8f7072c])
I20260924 19:58:09.951000 19467 tablet_loader.cc:96] loaded metadata for tablet accddc7f696142bfa6a5e655f5abc220 (table default.Table_bdae18e3011a421d96cdcc107e8f1dc2 [id=ca7affed79d94a2597fdaa1ff38e5e32])
I20260924 19:58:09.953351 19467 tablet_loader.cc:96] loaded metadata for tablet affce4de8e0b412aa3b4bf19f9be7b6e (table default.Table_25917a01f36140dfa4f504bc59aaf32b [id=a98fb79ea0af48b1bb3996ac782fcd06])
I20260924 19:58:09.954439 19467 tablet_loader.cc:96] loaded metadata for tablet b25dd98712d54cd5b53df00b2cf9f3c4 (table default.Table_e91b551877474bb0914b854a58447b3a [id=0606053a508145868b640f364fbf48d6])
I20260924 19:58:09.955391 19467 tablet_loader.cc:96] loaded metadata for tablet b474cde94649478b96809c7ef632f190 (table default.Table_b8cbd748636249c0ab17657b8f259893 [id=b9bee2849a334d098dd5d52fed7f6cc0])
I20260924 19:58:09.956826 19467 tablet_loader.cc:96] loaded metadata for tablet b7034626ce174d17bc82c92bfa6aebb3 (table default.Table_0a22025767fb456485c2624d5ca0dcbb [id=9f151548ce074cb6b8b8497fa246d61a])
I20260924 19:58:09.957863 19467 tablet_loader.cc:96] loaded metadata for tablet b7a1887d396f40a1a0c153dd0b2fe307 (table default.Table_954dba173a70455b90daee9230b44b6f [id=b743c3e3916a45069fabeed784e9e249])
W20260924 19:58:09.995728 17521 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.188:38761, user_credentials={real_user=slave}} blocked reactor thread for 69125.1us
W20260924 19:58:10.005918 17906 consensus_peers.cc:597] T ae7a0660795e4e4b87d0959ca84d5d1b P f74e6ed351284d7fb110e44336ce643a -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae7a0660795e4e4b87d0959ca84d5d1b. This is attempt 11: this message will repeat every 5th retry.
I20260924 19:58:10.024260 19467 tablet_loader.cc:96] loaded metadata for tablet bafaab10ce564b1b97659eccc0ab0967 (table default.Table_081e10d31dc74797abba37099d8a75fc [id=0ea22c5631c347aa9c7aa0e24140cb14])
I20260924 19:58:10.025599 19467 tablet_loader.cc:96] loaded metadata for tablet bea1ebdf647e4ef9a131a0398672b8ab (table default.Table_12bd04903f454058b69aa57c470e9ead [id=d2178935f68a4f2caef8377ad92221df])
I20260924 19:58:10.026888 19467 tablet_loader.cc:96] loaded metadata for tablet bf025253c49a4c23b79a25333cad160e (table default.Table_1d9dfc2ce03046ba955d3d82049402fd [id=c44dfe41598f4e6cbf320e3044b49fe5])
I20260924 19:58:10.029063 19467 tablet_loader.cc:96] loaded metadata for tablet bf2650dbcae54dcba04f0078ae6c3e1d (table default.Table_5128840101ca460b9e13dfd054764e9d [id=e8a1bc657d9b4fb2b1f0eb458d41dbbd])
I20260924 19:58:10.031543 19467 tablet_loader.cc:96] loaded metadata for tablet c0feecd2fdb4409f86f6a5fbfbab3537 (table default.Table_5e50f3dbbb674d658a5afac0ebcedaa6 [id=545488f05d324b748e42dae4a665f0d6])
I20260924 19:58:10.032969 19467 tablet_loader.cc:96] loaded metadata for tablet c1684aaf9d984649bff8c4b0b12044b9 (table default.Table_de89731e2fdb4f998ae007bd35b1a3f7 [id=f448bbbb806743b0b5232f59cf2367fe])
I20260924 19:58:10.035693 19467 tablet_loader.cc:96] loaded metadata for tablet ca37ddd22f5d4bcab2862553408447ce (table default.Table_ca51e47e4fc44d9ba5dd07cbeefa6e68 [id=6ea0f9084fd84e848fdecdd696cdc773])
I20260924 19:58:10.096989 19467 tablet_loader.cc:96] loaded metadata for tablet cba21f2e72914bc28a582433381a83f5 (table default.Table_9259880c400d44d69831fe68d31b06f4 [id=4f8c78c1109f4245b3ff41258543de1e])
I20260924 19:58:10.098156 19467 tablet_loader.cc:96] loaded metadata for tablet cfe0303596204e528585c4fe1c476d8b (table default.Table_bc465c5a40994196b5e0996ee2176340 [id=3b66c12b39cd46d9822c4d39e4a4a265])
I20260924 19:58:10.099261 19467 tablet_loader.cc:96] loaded metadata for tablet d129e0979a584f40ba9a975349cf15a5 (table default.Table_a12f8684de2041ec83f2fee77829afdd [id=94cea8de30ca4010a95f142ac8408653])
I20260924 19:58:10.101015 19467 tablet_loader.cc:96] loaded metadata for tablet d81f74c7ba2443e49228b17e867588cc (table default.Table_e3a6abe9c34a4fd6a8567a9c0a8190c3 [id=4e80d69486a34b848393f01e656d122b])
I20260924 19:58:10.109354 19467 tablet_loader.cc:96] loaded metadata for tablet d930e7f4a6b7471899dab1b28e40f93d (table default.Table_9b8f6ad547fc4c3fa379c93590aebd56 [id=4fc8e565a6a640549716cee596fa3f9d])
I20260924 19:58:10.115460 19467 tablet_loader.cc:96] loaded metadata for tablet dbb82914df65416aaf47db8e507afb40 (table default.Table_e2d6f648c21f44ad9674d28e00de881e [id=56af0067ee3548a29ee663929dfc9140])
I20260924 19:58:10.124068 19467 tablet_loader.cc:96] loaded metadata for tablet dd434e38c9364f55bd18383149015028 (table default.Table_a7d63063e7ee4493abe8c76da6d4f99d [id=be19c62729a34ae1a15966662303a869])
I20260924 19:58:10.125250 19467 tablet_loader.cc:96] loaded metadata for tablet df722b0bbfd04ce2943f0e5b8058c50f (table default.Table_db4a5e39fd954bb0b4c271899feefe90 [id=0b7e96c71a074f838a519746d9c92c3f])
I20260924 19:58:10.130582 19467 tablet_loader.cc:96] loaded metadata for tablet e0246d83189d4ba5b75fbd8390f517f6 (table default.Table_202c25f88e1a485c9ca6677d9cf53e75 [id=221fc6b9725747ada3ed73dc569052cb])
I20260924 19:58:10.139853 19467 tablet_loader.cc:96] loaded metadata for tablet e1137318a9fa4a0781f718dc34c4ecbe (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8])
I20260924 19:58:10.180867 19467 tablet_loader.cc:96] loaded metadata for tablet e59e737a68dd443f9898113b2b2a401e (table default.Table_1dad27c8fd4445fb89e32722eef0abe2 [id=4cd61b434f884eed908abf700d02cdd0])
I20260924 19:58:10.181998 19467 tablet_loader.cc:96] loaded metadata for tablet e68378312c334fc28dafb71bf6386be8 (table default.Table_8be3db507b2d42da90969bacc361c127 [id=cd5ff543cb1f48f58d5867aef9706963])
I20260924 19:58:10.183076 19467 tablet_loader.cc:96] loaded metadata for tablet edc6c16578e5463a9d44c25fd6c538a9 (table default.Table_88352246addd43f4be871c983413b2de [id=fa469a44fb3741d39036337129cc36a1])
I20260924 19:58:10.190447 19467 tablet_loader.cc:96] loaded metadata for tablet f10625ef84054ca0babbc583395dd96c (table default.Table_8ade2ea47f344fd18a687f7d5e0ea568 [id=eecec62f4e6a4e56920d841de1b9077b])
I20260924 19:58:10.191556 19467 tablet_loader.cc:96] loaded metadata for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803])
I20260924 19:58:10.195381 19467 tablet_loader.cc:96] loaded metadata for tablet f43f2a255c644ca6a68a8d867798fc1a (table default.Table_4e3a3158cde24302806b7ba7827831c4 [id=fa0b10df171340868b0305a28d192ea5])
I20260924 19:58:10.199047 19467 tablet_loader.cc:96] loaded metadata for tablet fbd5915ee98a4eda94479e9330cf8d04 (table default.Table_83714385573a44b2af65ac658fbe2cfa [id=e5b211f0af7143709eec620ecf04f2b5])
I20260924 19:58:10.200155 19467 tablet_loader.cc:96] loaded metadata for tablet fd6e656a653a43d1843a91fee813237d (table default.Table_e3179b11ea114ce99779ed0b2ac46732 [id=8c39c8f34038451d840db3c909d76f3d])
W20260924 19:58:10.231492 19467 catalog_manager.cc:1525] Time spent T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Loading table and tablet metadata into memory: real 18.567s user 0.320s sys 0.060s
I20260924 19:58:10.231787 19467 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260924 19:58:10.232201 17616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
W20260924 19:58:10.233407 17616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 11) took 13070 ms (client timeout 549 ms). Trace:
W20260924 19:58:10.233793 17616 rpcz_store.cc:269] 0924 19:57:57.163149 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:57.163394 (+ 245us) service_pool.cc:224] Handling call
0924 19:58:10.233291 (+13069897us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260924 19:58:10.235185 17616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 12) took 12507 ms (client timeout 549 ms). Trace:
W20260924 19:58:10.235473 17616 rpcz_store.cc:269] 0924 19:57:57.727884 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.234956 (+12507072us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.235107 (+ 151us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260924 19:58:10.236481 17623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
W20260924 19:58:10.237362 17623 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 5) took 17458 ms (client timeout 636 ms). Trace:
W20260924 19:58:10.237696 17623 rpcz_store.cc:269] 0924 19:57:52.779189 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:52.791385 (+ 12196us) service_pool.cc:224] Handling call
0924 19:58:10.237271 (+17445886us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:58:10.239921 17607 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:51280 (request call id 281) took 16697 ms. Trace:
I20260924 19:58:10.240263 17607 rpcz_store.cc:276] 0924 19:57:53.542885 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:53.543006 (+ 121us) service_pool.cc:224] Handling call
0924 19:58:10.239672 (+16696666us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:58:10.241173 17625 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
W20260924 19:58:10.241888 17625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 21) took 18357 ms (client timeout 527 ms). Trace:
I20260924 19:58:10.251223 19844 raft_consensus.cc:493] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:10.251704 17617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:58:10.251668 19844 raft_consensus.cc:515] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:58:10.254086 19844 leader_election.cc:290] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575), c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465)
W20260924 19:58:10.238945 17623 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 25) took 10942 ms (client timeout 712 ms). Trace:
W20260924 19:58:10.256479 17623 rpcz_store.cc:269] 0924 19:57:59.296084 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.238738 (+10942654us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.238864 (+ 126us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.257870 17906 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
W20260924 19:58:10.258932 17905 leader_election.cc:343] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): Not found: Tablet not found: ba46ac86b242407abd3e8dbf09f12efa
I20260924 19:58:10.259338 17905 leader_election.cc:304] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f74e6ed351284d7fb110e44336ce643a; no voters: c4f6e3a7b06148918d230c77ab0c8202, eaf64bfb64aa42948c9bc1411ce4ce35
I20260924 19:58:10.260002 19844 raft_consensus.cc:2749] T ba46ac86b242407abd3e8dbf09f12efa P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:58:10.247373 17625 rpcz_store.cc:269] 0924 19:57:51.884500 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:51.884633 (+ 133us) service_pool.cc:224] Handling call
0924 19:58:10.241813 (+18357180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:58:10.247090 17624 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 4 index: 322 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
W20260924 19:58:10.264258 17624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 22) took 17808 ms (client timeout 527 ms). Trace:
W20260924 19:58:10.264575 17624 rpcz_store.cc:269] 0924 19:57:52.455459 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:52.455605 (+ 146us) service_pool.cc:224] Handling call
0924 19:58:10.263971 (+17808366us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260924 19:58:10.274580 17617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 24) took 13578 ms (client timeout 712 ms). Trace:
I20260924 19:58:10.261912 17622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:58:10.278170 17619 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
W20260924 19:58:10.279388 17619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 10) took 14767 ms (client timeout 585 ms). Trace:
W20260924 19:58:10.279688 17619 rpcz_store.cc:269] 0924 19:57:55.511938 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:55.512145 (+ 207us) service_pool.cc:224] Handling call
0924 19:58:10.279318 (+14767173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:58:10.284927 19467 catalog_manager.cc:1308] Loaded cluster ID: 08019495472c40618d8cccdc88d885cc
I20260924 19:58:10.285240 19467 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
W20260924 19:58:10.286702 17619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 14) took 8725 ms (client timeout 531 ms). Trace:
W20260924 19:58:10.287005 17619 rpcz_store.cc:269] 0924 19:58:01.561854 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.286542 (+8724688us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.286651 (+ 109us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.275605 17617 rpcz_store.cc:269] 0924 19:57:56.696588 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:56.696762 (+ 174us) service_pool.cc:224] Handling call
0924 19:58:10.269152 (+13572390us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260924 19:58:10.289336 17623 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 26) took 10197 ms (client timeout 712 ms). Trace:
W20260924 19:58:10.289635 17623 rpcz_store.cc:269] 0924 19:58:00.080409 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.276320 (+10195911us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.277790 (+ 1470us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.290140 17616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 13) took 9343 ms (client timeout 531 ms). Trace:
W20260924 19:58:10.290457 17616 rpcz_store.cc:269] 0924 19:58:00.934685 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.276346 (+9341661us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.277916 (+ 1570us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.292032 17623 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 28) took 7048 ms (client timeout 603 ms). Trace:
W20260924 19:58:10.292341 17623 rpcz_store.cc:269] 0924 19:58:03.243847 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.291888 (+7048041us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.291980 (+ 92us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260924 19:58:10.288303 17618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:58:10.276577 17620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
W20260924 19:58:10.294101 17618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 23) took 14357 ms (client timeout 712 ms). Trace:
W20260924 19:58:10.294513 17618 rpcz_store.cc:269] 0924 19:57:55.937134 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:55.937323 (+ 189us) service_pool.cc:224] Handling call
0924 19:58:10.294042 (+14356719us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260924 19:58:10.295347 17618 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 16) took 6190 ms (client timeout 539 ms). Trace:
W20260924 19:58:10.295639 17618 rpcz_store.cc:269] 0924 19:58:04.105262 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.295197 (+6189935us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.295299 (+ 102us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.299342 17625 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 29) took 5684 ms (client timeout 688 ms). Trace:
W20260924 19:58:10.300325 17619 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 27) took 7704 ms (client timeout 603 ms). Trace:
W20260924 19:58:10.299810 17625 rpcz_store.cc:269] 0924 19:58:04.614746 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.299034 (+5684288us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.299254 (+ 220us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260924 19:58:10.302012 17621 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:53508 (request call id 8) took 15939 ms. Trace:
W20260924 19:58:10.302699 17617 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 30) took 4948 ms (client timeout 688 ms). Trace:
W20260924 19:58:10.302974 17617 rpcz_store.cc:269] 0924 19:58:05.351683 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.300072 (+4948389us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.300155 (+ 83us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260924 19:58:10.302314 17621 rpcz_store.cc:276] 0924 19:57:54.361793 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:54.362003 (+ 210us) service_pool.cc:224] Handling call
0924 19:58:10.300841 (+15938838us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260924 19:58:10.304499 17621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 17) took 4605 ms (client timeout 630 ms). Trace:
W20260924 19:58:10.304854 17621 rpcz_store.cc:269] 0924 19:58:05.699648 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.304346 (+4604698us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.304444 (+ 98us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.305807 17621 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 18) took 3931 ms (client timeout 630 ms). Trace:
W20260924 19:58:10.306083 17621 rpcz_store.cc:269] 0924 19:58:06.374480 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.305683 (+3931203us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.305765 (+ 82us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.308563 17624 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 31) took 3408 ms (client timeout 720 ms). Trace:
W20260924 19:58:10.308866 17624 rpcz_store.cc:269] 0924 19:58:06.900616 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.308426 (+3407810us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.308514 (+ 88us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.310595 17616 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 32) took 2643 ms (client timeout 720 ms). Trace:
W20260924 19:58:10.310940 17616 rpcz_store.cc:269] 0924 19:58:07.667379 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.310424 (+2643045us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.310529 (+ 105us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.310014 17620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 9) took 15423 ms (client timeout 585 ms). Trace:
W20260924 19:58:10.314029 17620 rpcz_store.cc:269] 0924 19:57:54.886200 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:54.886425 (+ 225us) service_pool.cc:224] Handling call
0924 19:58:10.309941 (+15423516us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260924 19:58:10.315323 17620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 19) took 2329 ms (client timeout 684 ms). Trace:
W20260924 19:58:10.315640 17620 rpcz_store.cc:269] 0924 19:58:07.985536 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.315173 (+2329637us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.315265 (+ 92us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.317641 17620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 20) took 1623 ms (client timeout 684 ms). Trace:
W20260924 19:58:10.317915 17620 rpcz_store.cc:269] 0924 19:58:08.694197 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.317503 (+1623306us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.317592 (+ 89us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.318776 17622 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 6) took 16910 ms (client timeout 636 ms). Trace:
W20260924 19:58:10.319717 17620 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38988 (request call id 33) took 942 ms (client timeout 547 ms). Trace:
W20260924 19:58:10.320000 17620 rpcz_store.cc:269] 0924 19:58:09.377199 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.319586 (+942387us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.319672 (+ 86us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.319098 17622 rpcz_store.cc:269] 0924 19:57:53.408859 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:57:53.409060 (+ 201us) service_pool.cc:224] Handling call
0924 19:58:10.318705 (+16909645us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260924 19:58:10.323693 17622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:58:10.329284 19467 catalog_manager.cc:1553] Loading token signing keys...
W20260924 19:58:10.329489 17623 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53508 (request call id 15) took 6758 ms (client timeout 539 ms). Trace:
W20260924 19:58:10.329805 17623 rpcz_store.cc:269] 0924 19:58:03.535381 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.293816 (+6758435us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.293899 (+ 83us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260924 19:58:10.330571 17619 rpcz_store.cc:269] 0924 19:58:02.588385 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:10.288045 (+7699660us) service_pool.cc:211] Skipping call since client already timed out
0924 19:58:10.292904 (+ 4859us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260924 19:58:10.332863 17521 leader_election.cc:304] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6f88e00b235f48bb82e6a3c9c9f51fa6; no voters: 3e059bbe4dac4178bccb3be48cb1118d, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:58:10.345511 19847 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6f88e00b235f48bb82e6a3c9c9f51fa6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 19:58:10.366181 19467 catalog_manager.cc:6230] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Loaded TSK: 0
I20260924 19:58:10.367348 19467 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260924 19:58:10.405957 19848 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e1ff67c5d57d474a9d92cc4c6de1287e)
W20260924 19:58:10.408115 18051 consensus_peers.cc:597] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57008266b80345e0a3f8e60bd7e2d62b. This is attempt 11: this message will repeat every 5th retry.
I20260924 19:58:10.411911 19848 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e059bbe4dac4178bccb3be48cb1118d" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 46309 } } peers { permanent_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 43689 } } peers { permanent_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 38761 } }
I20260924 19:58:10.415216 19848 leader_election.cc:290] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6f88e00b235f48bb82e6a3c9c9f51fa6 (127.17.2.189:43689), e1ff67c5d57d474a9d92cc4c6de1287e (127.17.2.188:38761)
I20260924 19:58:10.415092 17536 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "6f88e00b235f48bb82e6a3c9c9f51fa6" is_pre_election: true
I20260924 19:58:10.416065 17620 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e059bbe4dac4178bccb3be48cb1118d" candidate_term: 6 candidate_status { last_received { term: 5 index: 323 } } ignore_live_leader: false dest_uuid: "e1ff67c5d57d474a9d92cc4c6de1287e" is_pre_election: true
I20260924 19:58:10.417418 18903 leader_election.cc:304] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e059bbe4dac4178bccb3be48cb1118d; no voters: 6f88e00b235f48bb82e6a3c9c9f51fa6, e1ff67c5d57d474a9d92cc4c6de1287e
I20260924 19:58:10.419616 19848 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 3e059bbe4dac4178bccb3be48cb1118d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260924 19:58:10.478909 17607 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_0588d141fbec45f2a42b728f4c01e32c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:10.479835 17614 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_74c0905eab544ac1966d90b9d3f3f548"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:58:10.481258 17614 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74c0905eab544ac1966d90b9d3f3f548 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:10.481215 17607 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0588d141fbec45f2a42b728f4c01e32c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:10.508229 17615 master_service.cc:960] ReplaceTablet: received request to replace tablet accddc7f696142bfa6a5e655f5abc220 from {username='slave'} at 127.0.0.1:51280
I20260924 19:58:10.518545 17609 master_service.cc:960] ReplaceTablet: received request to replace tablet 484517ee7df8463bb70d201ead1c6835 from {username='slave'} at 127.0.0.1:51280
I20260924 19:58:10.600935 17613 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_102d9262505c41738c4c97fc11551bcb" } modify_external_catalogs: true
I20260924 19:58:10.601550 17613 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51280:
table { table_name: "default.Table_102d9262505c41738c4c97fc11551bcb" } modify_external_catalogs: true
I20260924 19:58:10.604156 17610 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_4450908a8c774e0080f55e1df579fb27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:58:10.605669 17610 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4450908a8c774e0080f55e1df579fb27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:10.619354 17611 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51280:
name: "default.Table_06638fcce4cf4f07abe9f199ff4eea8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:58:10.620913 17611 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06638fcce4cf4f07abe9f199ff4eea8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:10.647811 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 57008266b80345e0a3f8e60bd7e2d62b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:57:37 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:58:10.664503 19856 tablet_replica.cc:333] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:58:10.719365 19856 raft_consensus.cc:2243] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:10.720655 19856 raft_consensus.cc:2272] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:10.724085 19856 ts_tablet_manager.cc:1916] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:10.845935 19856 ts_tablet_manager.cc:1929] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:10.854592 19856 log.cc:1199] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/57008266b80345e0a3f8e60bd7e2d62b
I20260924 19:58:10.855749 19856 ts_tablet_manager.cc:1950] T 57008266b80345e0a3f8e60bd7e2d62b P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:58:10.864713 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 57008266b80345e0a3f8e60bd7e2d62b (table default.Table_cca06980f109487a9ed8b8172950b60d [id=016f4039733144c985808b862eebcc9a]) successfully deleted
W20260924 19:58:10.907872 17906 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:42465, user_credentials={real_user=slave}} blocked reactor thread for 61839.3us
I20260924 19:58:10.912496 17615 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet accddc7f696142bfa6a5e655f5abc220
I20260924 19:58:10.913985 17615 catalog_manager.cc:6934] ReplaceTablet: tablet accddc7f696142bfa6a5e655f5abc220 deleted and replaced by tablet 402312c0cda443bbbab576b482183b58
I20260924 19:58:10.916093 17795 tablet_service.cc:1558] Processing DeleteTablet for tablet accddc7f696142bfa6a5e655f5abc220 with delete_type TABLET_DATA_DELETED (replaced by tablet 402312c0cda443bbbab576b482183b58) from {username='slave'} at 127.0.0.1:36562
I20260924 19:58:10.919147 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet accddc7f696142bfa6a5e655f5abc220 with delete_type TABLET_DATA_DELETED (replaced by tablet 402312c0cda443bbbab576b482183b58) from {username='slave'} at 127.0.0.1:38450
I20260924 19:58:10.919857 19856 tablet_replica.cc:333] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:58:10.920729 19856 raft_consensus.cc:2243] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:10.921214 19856 raft_consensus.cc:2272] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:10.923213 19856 ts_tablet_manager.cc:1916] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:10.927788 17609 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 484517ee7df8463bb70d201ead1c6835
I20260924 19:58:10.929230 17609 catalog_manager.cc:6934] ReplaceTablet: tablet 484517ee7df8463bb70d201ead1c6835 deleted and replaced by tablet 5fdde2770dc4494189453ba4f01c4e3a
I20260924 19:58:10.936455 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 484517ee7df8463bb70d201ead1c6835 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fdde2770dc4494189453ba4f01c4e3a) from {username='slave'} at 127.0.0.1:36562
I20260924 19:58:10.941745 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 484517ee7df8463bb70d201ead1c6835 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fdde2770dc4494189453ba4f01c4e3a) from {username='slave'} at 127.0.0.1:38450
I20260924 19:58:10.950124 18017 heartbeater.cc:499] Master 127.17.2.188:38761 was elected leader, sending a full tablet report...
I20260924 19:58:10.953078 17613 catalog_manager.cc:6133] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: Sending DeleteTablet for 3 replicas of tablet 077f70bcee7e412299da12ddc30a63ff
I20260924 19:58:10.956218 17800 tablet_service.cc:1558] Processing DeleteTablet for tablet 077f70bcee7e412299da12ddc30a63ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:10 UTC) from {username='slave'} at 127.0.0.1:36562
I20260924 19:58:10.958787 18086 tablet_service.cc:1558] Processing DeleteTablet for tablet 077f70bcee7e412299da12ddc30a63ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:10 UTC) from {username='slave'} at 127.0.0.1:38450
I20260924 19:58:10.959880 19856 ts_tablet_manager.cc:1929] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:10.960242 19856 log.cc:1199] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/accddc7f696142bfa6a5e655f5abc220
I20260924 19:58:10.960961 19856 ts_tablet_manager.cc:1950] T accddc7f696142bfa6a5e655f5abc220 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:58:10.963673 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet accddc7f696142bfa6a5e655f5abc220 (table default.Table_bdae18e3011a421d96cdcc107e8f1dc2 [id=ca7affed79d94a2597fdaa1ff38e5e32]) successfully deleted
I20260924 19:58:10.987733 19856 tablet_replica.cc:333] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:58:10.988662 19856 raft_consensus.cc:2243] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:10.989571 19856 raft_consensus.cc:2272] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:10.991647 19856 ts_tablet_manager.cc:1916] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:10.992259 19865 tablet_replica.cc:333] T accddc7f696142bfa6a5e655f5abc220 P c4f6e3a7b06148918d230c77ab0c8202: stopping tablet replica
I20260924 19:58:11.008579 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet accddc7f696142bfa6a5e655f5abc220 with delete_type TABLET_DATA_DELETED (replaced by tablet 402312c0cda443bbbab576b482183b58) from {username='slave'} at 127.0.0.1:51332
I20260924 19:58:11.016659 17418 master-stress-test.cc:581] Killed test thread 1/10
I20260924 19:58:11.017047 17418 master-stress-test.cc:579] Killing test thread 2/10
I20260924 19:58:11.017289 17418 master-stress-test.cc:581] Killed test thread 2/10
I20260924 19:58:11.017483 17418 master-stress-test.cc:579] Killing test thread 3/10
I20260924 19:58:11.017693 17418 master-stress-test.cc:581] Killed test thread 3/10
I20260924 19:58:11.017886 17418 master-stress-test.cc:579] Killing test thread 4/10
I20260924 19:58:11.018105 17418 master-stress-test.cc:581] Killed test thread 4/10
I20260924 19:58:11.018038 19869 tablet_replica.cc:333] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:58:11.018290 17418 master-stress-test.cc:579] Killing test thread 5/10
I20260924 19:58:11.018513 17418 master-stress-test.cc:581] Killed test thread 5/10
I20260924 19:58:11.018699 17418 master-stress-test.cc:579] Killing test thread 6/10
I20260924 19:58:11.019063 19869 raft_consensus.cc:2243] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:11.019410 17418 master-stress-test.cc:581] Killed test thread 6/10
I20260924 19:58:11.019635 17418 master-stress-test.cc:579] Killing test thread 7/10
I20260924 19:58:11.019958 17418 master-stress-test.cc:581] Killed test thread 7/10
I20260924 19:58:11.020161 17418 master-stress-test.cc:579] Killing test thread 8/10
I20260924 19:58:11.020416 19869 raft_consensus.cc:2272] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:11.020507 17418 master-stress-test.cc:581] Killed test thread 8/10
I20260924 19:58:11.020694 17418 master-stress-test.cc:579] Killing test thread 9/10
I20260924 19:58:11.020949 17418 master-stress-test.cc:581] Killed test thread 9/10
I20260924 19:58:11.021136 17418 master-stress-test.cc:579] Killing test thread 10/10
I20260924 19:58:11.021430 17418 master-stress-test.cc:581] Killed test thread 10/10
I20260924 19:58:11.021767 17418 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu with pid 17714
I20260924 19:58:11.023844 19869 ts_tablet_manager.cc:1916] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:11.030689 17945 tablet_service.cc:1558] Processing DeleteTablet for tablet 077f70bcee7e412299da12ddc30a63ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:10 UTC) from {username='slave'} at 127.0.0.1:51332
I20260924 19:58:11.031587 17940 tablet_service.cc:1558] Processing DeleteTablet for tablet 484517ee7df8463bb70d201ead1c6835 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fdde2770dc4494189453ba4f01c4e3a) from {username='slave'} at 127.0.0.1:51332
I20260924 19:58:11.045738 19869 ts_tablet_manager.cc:1929] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:11.046520 19869 log.cc:1199] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/accddc7f696142bfa6a5e655f5abc220
I20260924 19:58:11.056006 19856 ts_tablet_manager.cc:1929] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:11.056411 19856 log.cc:1199] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/484517ee7df8463bb70d201ead1c6835
I20260924 19:58:11.057098 19856 ts_tablet_manager.cc:1950] T 484517ee7df8463bb70d201ead1c6835 P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:58:11.057401 19869 ts_tablet_manager.cc:1950] T accddc7f696142bfa6a5e655f5abc220 P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:58:11.058883 19869 tablet_replica.cc:333] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a: stopping tablet replica
I20260924 19:58:11.059487 17599 catalog_manager.cc:5063] TS f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): tablet accddc7f696142bfa6a5e655f5abc220 (table default.Table_bdae18e3011a421d96cdcc107e8f1dc2 [id=ca7affed79d94a2597fdaa1ff38e5e32]) successfully deleted
I20260924 19:58:11.074657 19869 raft_consensus.cc:2243] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:11.075341 19869 raft_consensus.cc:2272] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:11.077576 19869 ts_tablet_manager.cc:1916] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:11.093036 19856 tablet_replica.cc:333] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35: stopping tablet replica
I20260924 19:58:11.093248 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 484517ee7df8463bb70d201ead1c6835 (table default.Table_771765c449ce492986addb028e1b11d1 [id=f954ad85289641fda971e5c5155b9147]) successfully deleted
I20260924 19:58:11.093868 19856 raft_consensus.cc:2243] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:11.094897 19856 raft_consensus.cc:2272] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:11.097508 19856 ts_tablet_manager.cc:1916] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:11.120790 19856 ts_tablet_manager.cc:1929] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:58:11.121452 19856 log.cc:1199] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/077f70bcee7e412299da12ddc30a63ff
I20260924 19:58:11.122148 19856 ts_tablet_manager.cc:1950] T 077f70bcee7e412299da12ddc30a63ff P eaf64bfb64aa42948c9bc1411ce4ce35: Deleting consensus metadata
I20260924 19:58:11.124791 17599 catalog_manager.cc:5063] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575): tablet 077f70bcee7e412299da12ddc30a63ff (table default.Table_102d9262505c41738c4c97fc11551bcb [id=3ee2ed8b290747128a29f7c107dbfe82]) successfully deleted
I20260924 19:58:11.147365 18163 heartbeater.cc:499] Master 127.17.2.188:38761 was elected leader, sending a full tablet report...
I20260924 19:58:11.154295 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 56295c0109ce489fb03d65674df62c44 (DEFAULT_TABLE table=default.Table_b7e4d07630f24ee0ab58d5269687448d [id=f9f9606f84804c769a867198983fb713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.155473 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56295c0109ce489fb03d65674df62c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.160403 18082 tablet_service.cc:1511] Processing CreateTablet for tablet f10625ef84054ca0babbc583395dd96c (DEFAULT_TABLE table=default.Table_8ade2ea47f344fd18a687f7d5e0ea568 [id=eecec62f4e6a4e56920d841de1b9077b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.161414 18082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10625ef84054ca0babbc583395dd96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.164373 18095 tablet_service.cc:1511] Processing CreateTablet for tablet 3e52ecf08c6c45d4b8ba423358fb5fc1 (DEFAULT_TABLE table=default.Table_dea06619a5364df0961e47cb617c9e52 [id=e40a55e03a2d4ceea51d555d0c60a334]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.165293 18095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e52ecf08c6c45d4b8ba423358fb5fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.168429 18091 tablet_service.cc:1511] Processing CreateTablet for tablet 28fb29faeeb34c57bb6e52a9c5ffd218 (DEFAULT_TABLE table=default.Table_843b88f433f044458daa98228935a46e [id=c19508f168ed4191bdd25de1107ab416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.169453 18091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28fb29faeeb34c57bb6e52a9c5ffd218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.172976 18090 tablet_service.cc:1511] Processing CreateTablet for tablet b474cde94649478b96809c7ef632f190 (DEFAULT_TABLE table=default.Table_b8cbd748636249c0ab17657b8f259893 [id=b9bee2849a334d098dd5d52fed7f6cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.174016 18090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b474cde94649478b96809c7ef632f190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.175904 18084 tablet_service.cc:1511] Processing CreateTablet for tablet bf2650dbcae54dcba04f0078ae6c3e1d (DEFAULT_TABLE table=default.Table_5128840101ca460b9e13dfd054764e9d [id=e8a1bc657d9b4fb2b1f0eb458d41dbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.176796 18084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2650dbcae54dcba04f0078ae6c3e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.179096 18088 tablet_service.cc:1511] Processing CreateTablet for tablet 8ea2385391d34f8abd9d105fbd7f22cf (DEFAULT_TABLE table=default.Table_a36de9161f8f4ea08243a7222c099822 [id=ff9a4ba6b88648748c228135ed440066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.180142 18088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ea2385391d34f8abd9d105fbd7f22cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.182847 18097 tablet_service.cc:1511] Processing CreateTablet for tablet b7034626ce174d17bc82c92bfa6aebb3 (DEFAULT_TABLE table=default.Table_0a22025767fb456485c2624d5ca0dcbb [id=9f151548ce074cb6b8b8497fa246d61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.183871 18097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7034626ce174d17bc82c92bfa6aebb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.192682 18085 tablet_service.cc:1511] Processing CreateTablet for tablet 9936ffa261ea4aa98426b00a4ef3c6a9 (DEFAULT_TABLE table=default.Table_b6d3fa192eb741158adf5b69323a29de [id=9ead6eed50694fd1b920b5b0d46891c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.193871 18085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9936ffa261ea4aa98426b00a4ef3c6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.196378 18098 tablet_service.cc:1511] Processing CreateTablet for tablet 46e703a4892f445bb0a1c9f5a46db63d (DEFAULT_TABLE table=default.Table_ee93a1803fd54a4ba6723d1e3783c004 [id=fec7f8aa891e42c1a84fe32fe4e0c44f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.197384 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e703a4892f445bb0a1c9f5a46db63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.200090 18089 tablet_service.cc:1511] Processing CreateTablet for tablet 66599c38b1a64ad28fe7bd8e5b5056a8 (DEFAULT_TABLE table=default.Table_a50ccb6bd6a042bc99b503e5aa1188ad [id=bfd5e6e55fb049f9965b190374b38c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.201167 18089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66599c38b1a64ad28fe7bd8e5b5056a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.204051 18087 tablet_service.cc:1511] Processing CreateTablet for tablet 17973e8edf1a41f79de5ace4f4beedbf (DEFAULT_TABLE table=default.Table_0588d141fbec45f2a42b728f4c01e32c [id=08d899012a664586ac8fbcc2e35cab7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.205148 18087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17973e8edf1a41f79de5ace4f4beedbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.208092 18092 tablet_service.cc:1511] Processing CreateTablet for tablet 83cfc9998eb84e66b61e88b010992e3e (DEFAULT_TABLE table=default.Table_0c6b7fd4e7bd42a3ba0d407aea438273 [id=79e6d48b52b740958363f06cf05f2f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.209147 18092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83cfc9998eb84e66b61e88b010992e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.212909 18096 tablet_service.cc:1511] Processing CreateTablet for tablet 2b0b011357674a42a09d83f9384abaf1 (DEFAULT_TABLE table=default.Table_5aeda90fd8e3425a8c49a239533690c9 [id=63622937f5a64a05b56aa1310407ec99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.213078 18083 tablet_service.cc:1511] Processing CreateTablet for tablet 3eb3d893415b4dfcbcacd435c3c16e5e (DEFAULT_TABLE table=default.Table_c42c4d9b6c1e4d2abd6aa51f251be773 [id=a72d39e753b2435a800e68d308cdc86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.214396 18093 tablet_service.cc:1511] Processing CreateTablet for tablet cba21f2e72914bc28a582433381a83f5 (DEFAULT_TABLE table=default.Table_9259880c400d44d69831fe68d31b06f4 [id=4f8c78c1109f4245b3ff41258543de1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.216063 18096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0b011357674a42a09d83f9384abaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.218628 18081 tablet_service.cc:1511] Processing CreateTablet for tablet 492eb083f50e40a388d2174d6dc8445b (DEFAULT_TABLE table=default.Table_9faf28b24a2c47449a862536ea873f36 [id=3d02208bd3604c729b474a4f005b8cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.219846 18083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eb3d893415b4dfcbcacd435c3c16e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.223893 18081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 492eb083f50e40a388d2174d6dc8445b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.227674 18093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba21f2e72914bc28a582433381a83f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.300112 18079 tablet_service.cc:1511] Processing CreateTablet for tablet c81c5fd07c9649f5aecbfef4c4dfefbd (DEFAULT_TABLE table=default.Table_4450908a8c774e0080f55e1df579fb27 [id=05deca50714d4def9b7d4657fa5ff9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.301390 18079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81c5fd07c9649f5aecbfef4c4dfefbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.305094 17945 tablet_service.cc:1511] Processing CreateTablet for tablet f10625ef84054ca0babbc583395dd96c (DEFAULT_TABLE table=default.Table_8ade2ea47f344fd18a687f7d5e0ea568 [id=eecec62f4e6a4e56920d841de1b9077b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.307050 17950 tablet_service.cc:1511] Processing CreateTablet for tablet b474cde94649478b96809c7ef632f190 (DEFAULT_TABLE table=default.Table_b8cbd748636249c0ab17657b8f259893 [id=b9bee2849a334d098dd5d52fed7f6cc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.309613 17940 tablet_service.cc:1511] Processing CreateTablet for tablet 56295c0109ce489fb03d65674df62c44 (DEFAULT_TABLE table=default.Table_b7e4d07630f24ee0ab58d5269687448d [id=f9f9606f84804c769a867198983fb713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.311496 17941 tablet_service.cc:1511] Processing CreateTablet for tablet 3e52ecf08c6c45d4b8ba423358fb5fc1 (DEFAULT_TABLE table=default.Table_dea06619a5364df0961e47cb617c9e52 [id=e40a55e03a2d4ceea51d555d0c60a334]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.313282 17944 tablet_service.cc:1511] Processing CreateTablet for tablet 8ea2385391d34f8abd9d105fbd7f22cf (DEFAULT_TABLE table=default.Table_a36de9161f8f4ea08243a7222c099822 [id=ff9a4ba6b88648748c228135ed440066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.315058 17947 tablet_service.cc:1511] Processing CreateTablet for tablet b7034626ce174d17bc82c92bfa6aebb3 (DEFAULT_TABLE table=default.Table_0a22025767fb456485c2624d5ca0dcbb [id=9f151548ce074cb6b8b8497fa246d61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.316871 17948 tablet_service.cc:1511] Processing CreateTablet for tablet 28fb29faeeb34c57bb6e52a9c5ffd218 (DEFAULT_TABLE table=default.Table_843b88f433f044458daa98228935a46e [id=c19508f168ed4191bdd25de1107ab416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.318624 17949 tablet_service.cc:1511] Processing CreateTablet for tablet bf2650dbcae54dcba04f0078ae6c3e1d (DEFAULT_TABLE table=default.Table_5128840101ca460b9e13dfd054764e9d [id=e8a1bc657d9b4fb2b1f0eb458d41dbbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.324467 17946 tablet_service.cc:1511] Processing CreateTablet for tablet 9936ffa261ea4aa98426b00a4ef3c6a9 (DEFAULT_TABLE table=default.Table_b6d3fa192eb741158adf5b69323a29de [id=9ead6eed50694fd1b920b5b0d46891c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.336872 17945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10625ef84054ca0babbc583395dd96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.344931 18094 tablet_service.cc:1511] Processing CreateTablet for tablet 5fdde2770dc4494189453ba4f01c4e3a (DEFAULT_TABLE table=default.Table_771765c449ce492986addb028e1b11d1 [id=f954ad85289641fda971e5c5155b9147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.346462 18094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fdde2770dc4494189453ba4f01c4e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.366809 17950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b474cde94649478b96809c7ef632f190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.381969 17943 tablet_service.cc:1511] Processing CreateTablet for tablet 66599c38b1a64ad28fe7bd8e5b5056a8 (DEFAULT_TABLE table=default.Table_a50ccb6bd6a042bc99b503e5aa1188ad [id=bfd5e6e55fb049f9965b190374b38c3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.382637 17942 tablet_service.cc:1511] Processing CreateTablet for tablet 46e703a4892f445bb0a1c9f5a46db63d (DEFAULT_TABLE table=default.Table_ee93a1803fd54a4ba6723d1e3783c004 [id=fec7f8aa891e42c1a84fe32fe4e0c44f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.384717 17951 tablet_service.cc:1511] Processing CreateTablet for tablet 17973e8edf1a41f79de5ace4f4beedbf (DEFAULT_TABLE table=default.Table_0588d141fbec45f2a42b728f4c01e32c [id=08d899012a664586ac8fbcc2e35cab7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.390097 17952 tablet_service.cc:1511] Processing CreateTablet for tablet 83cfc9998eb84e66b61e88b010992e3e (DEFAULT_TABLE table=default.Table_0c6b7fd4e7bd42a3ba0d407aea438273 [id=79e6d48b52b740958363f06cf05f2f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.392354 17939 tablet_service.cc:1511] Processing CreateTablet for tablet 2b0b011357674a42a09d83f9384abaf1 (DEFAULT_TABLE table=default.Table_5aeda90fd8e3425a8c49a239533690c9 [id=63622937f5a64a05b56aa1310407ec99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.396652 17938 tablet_service.cc:1511] Processing CreateTablet for tablet cba21f2e72914bc28a582433381a83f5 (DEFAULT_TABLE table=default.Table_9259880c400d44d69831fe68d31b06f4 [id=4f8c78c1109f4245b3ff41258543de1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.400213 17937 tablet_service.cc:1511] Processing CreateTablet for tablet 3eb3d893415b4dfcbcacd435c3c16e5e (DEFAULT_TABLE table=default.Table_c42c4d9b6c1e4d2abd6aa51f251be773 [id=a72d39e753b2435a800e68d308cdc86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.403699 17936 tablet_service.cc:1511] Processing CreateTablet for tablet 492eb083f50e40a388d2174d6dc8445b (DEFAULT_TABLE table=default.Table_9faf28b24a2c47449a862536ea873f36 [id=3d02208bd3604c729b474a4f005b8cf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.411809 18080 tablet_service.cc:1511] Processing CreateTablet for tablet 81e51fb00b4d4ecf89b0a83e8ee0f799 (DEFAULT_TABLE table=default.Table_06638fcce4cf4f07abe9f199ff4eea8f [id=e7ffe893a1dc49ed81b8d4aa13970766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.413087 18080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e51fb00b4d4ecf89b0a83e8ee0f799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.445746 17940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56295c0109ce489fb03d65674df62c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.449159 17935 tablet_service.cc:1511] Processing CreateTablet for tablet c81c5fd07c9649f5aecbfef4c4dfefbd (DEFAULT_TABLE table=default.Table_4450908a8c774e0080f55e1df579fb27 [id=05deca50714d4def9b7d4657fa5ff9a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.455080 17933 tablet_service.cc:1511] Processing CreateTablet for tablet 81e51fb00b4d4ecf89b0a83e8ee0f799 (DEFAULT_TABLE table=default.Table_06638fcce4cf4f07abe9f199ff4eea8f [id=e7ffe893a1dc49ed81b8d4aa13970766]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.463464 17934 tablet_service.cc:1511] Processing CreateTablet for tablet 5fdde2770dc4494189453ba4f01c4e3a (DEFAULT_TABLE table=default.Table_771765c449ce492986addb028e1b11d1 [id=f954ad85289641fda971e5c5155b9147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.468758 17941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e52ecf08c6c45d4b8ba423358fb5fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.473569 17944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ea2385391d34f8abd9d105fbd7f22cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.474323 17935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c81c5fd07c9649f5aecbfef4c4dfefbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.475765 17947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7034626ce174d17bc82c92bfa6aebb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.493052 18098 tablet_service.cc:1511] Processing CreateTablet for tablet bc67f182fdaf43c785a7af0152ff2277 (DEFAULT_TABLE table=default.Table_74c0905eab544ac1966d90b9d3f3f548 [id=6e93eeaa41f244a2a6e70492b2ec31fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.501245 18086 tablet_service.cc:1511] Processing CreateTablet for tablet 402312c0cda443bbbab576b482183b58 (DEFAULT_TABLE table=default.Table_bdae18e3011a421d96cdcc107e8f1dc2 [id=ca7affed79d94a2597fdaa1ff38e5e32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:11.513007 19873 tablet_bootstrap.cc:492] T 46e703a4892f445bb0a1c9f5a46db63d P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
I20260924 19:58:11.530257 17946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9936ffa261ea4aa98426b00a4ef3c6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.531385 17949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2650dbcae54dcba04f0078ae6c3e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.537374 17948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28fb29faeeb34c57bb6e52a9c5ffd218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.540812 17951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17973e8edf1a41f79de5ace4f4beedbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.544697 18098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc67f182fdaf43c785a7af0152ff2277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.544570 19871 raft_consensus.cc:493] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:11.544584 17952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83cfc9998eb84e66b61e88b010992e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.554388 18086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 402312c0cda443bbbab576b482183b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.559580 17934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fdde2770dc4494189453ba4f01c4e3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.561673 17938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba21f2e72914bc28a582433381a83f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.568060 17933 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e51fb00b4d4ecf89b0a83e8ee0f799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.570066 17936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 492eb083f50e40a388d2174d6dc8445b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:11.545235 19871 raft_consensus.cc:515] T 44199d2655f14dcf97afe2180789aae4 P f74e6ed351284d7fb110e44336ce643a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } }
I20260924 19:58:11.590461 19869 ts_tablet_manager.cc:1929] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:58:11.591066 19869 log.cc:1199] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/077f70bcee7e412299da12ddc30a63ff
I20260924 19:58:11.592042 19869 ts_tablet_manager.cc:1950] T 077f70bcee7e412299da12ddc30a63ff P f74e6ed351284d7fb110e44336ce643a: Deleting consensus metadata
I20260924 19:58:11.599453 17939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0b011357674a42a09d83f9384abaf1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:58:11.620033 17600 connection.cc:573] client connection to 127.17.2.129:42465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:58:11.620131 18051 connection.cc:573] client connection to 127.17.2.129:42465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:58:11.620478 17600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 2 similar messages]
W20260924 19:58:11.620605 18051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:58:11.632771 18050 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:41301, user_credentials={real_user=slave}} blocked reactor thread for 49331.5us
I20260924 19:58:11.649749 17418 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu with pid 17877
W20260924 19:58:11.657368 17600 catalog_manager.cc:4790] TS c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet accddc7f696142bfa6a5e655f5abc220: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111)
I20260924 19:58:11.659003 19873 tablet_bootstrap.cc:654] T 46e703a4892f445bb0a1c9f5a46db63d P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:58:11.676426 18051 consensus_peers.cc:597] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.676893 18051 consensus_peers.cc:597] T dbb82914df65416aaf47db8e507afb40 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.677248 18051 consensus_peers.cc:597] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.677551 18051 consensus_peers.cc:597] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.677877 18051 consensus_peers.cc:597] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.678433 18051 consensus_peers.cc:597] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.678902 18051 consensus_peers.cc:597] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.679317 18051 consensus_peers.cc:597] T e68378312c334fc28dafb71bf6386be8 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.679670 18051 consensus_peers.cc:597] T 872bc32b9e384c828a0699079b27530d P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.679988 18051 consensus_peers.cc:597] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.696589 18050 connection.cc:573] server connection from 127.17.2.129:42273 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:58:11.713292 19873 tablet_bootstrap.cc:492] T 46e703a4892f445bb0a1c9f5a46db63d P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:58:11.717351 19873 ts_tablet_manager.cc:1403] T 46e703a4892f445bb0a1c9f5a46db63d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.205s user 0.004s sys 0.011s
W20260924 19:58:11.714095 18051 consensus_peers.cc:597] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:11.720711 19873 raft_consensus.cc:359] T 46e703a4892f445bb0a1c9f5a46db63d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:58:11.721629 18051 consensus_peers.cc:597] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.722086 18051 consensus_peers.cc:597] T 6bd442f16492495f84bb8363b9bade97 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.722415 18051 consensus_peers.cc:597] T affce4de8e0b412aa3b4bf19f9be7b6e P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.722728 18051 consensus_peers.cc:597] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:11.721676 19873 raft_consensus.cc:385] T 46e703a4892f445bb0a1c9f5a46db63d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:11.723220 19873 raft_consensus.cc:740] T 46e703a4892f445bb0a1c9f5a46db63d P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:58:11.725150 19873 consensus_queue.cc:260] T 46e703a4892f445bb0a1c9f5a46db63d P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
W20260924 19:58:11.737267 18051 consensus_peers.cc:597] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.737666 18051 consensus_peers.cc:597] T cb111f7a189c4f00a41686d647c64e65 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.738006 18051 consensus_peers.cc:597] T f43f2a255c644ca6a68a8d867798fc1a P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.738299 18051 consensus_peers.cc:597] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.738601 18051 consensus_peers.cc:597] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.738974 18051 consensus_peers.cc:597] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.746889 18051 consensus_peers.cc:597] T 0c6943a365da464e83cedc2b9711ca89 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.747418 18051 consensus_peers.cc:597] T e0246d83189d4ba5b75fbd8390f517f6 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.747756 18051 consensus_peers.cc:597] T 0c08f10d2a364cb8a0a7ea40654c3fe4 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.748445 18051 consensus_peers.cc:597] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.748816 18051 consensus_peers.cc:597] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.749130 18051 consensus_peers.cc:597] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.749547 18051 consensus_peers.cc:597] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.749867 18051 consensus_peers.cc:597] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.750391 18051 consensus_peers.cc:597] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.750737 18051 consensus_peers.cc:597] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:11.758805 19873 ts_tablet_manager.cc:1434] T 46e703a4892f445bb0a1c9f5a46db63d P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
W20260924 19:58:11.759697 18051 consensus_peers.cc:597] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:11.764254 19873 tablet_bootstrap.cc:492] T 66599c38b1a64ad28fe7bd8e5b5056a8 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:58:11.774999 18051 consensus_peers.cc:597] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:11.780037 17610 catalog_manager.cc:5838] T df722b0bbfd04ce2943f0e5b8058c50f P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.781673 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet df722b0bbfd04ce2943f0e5b8058c50f (table default.Table_db4a5e39fd954bb0b4c271899feefe90 [id=0b7e96c71a074f838a519746d9c92c3f]). Expected version 1 got 0
I20260924 19:58:11.782480 17610 catalog_manager.cc:5838] T 78801b8d0b664217a74b20160ba2d8ac P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 2, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 2 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
I20260924 19:58:11.786034 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 78801b8d0b664217a74b20160ba2d8ac (table default.Table_454683a3d79d4988b41292f7776df549 [id=81b26242949442c0a7db6f55f0f18097]). Expected version 1 got 0
W20260924 19:58:11.788622 18051 consensus_peers.cc:597] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:11.791610 17610 catalog_manager.cc:5838] T cfe0303596204e528585c4fe1c476d8b P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.794018 17610 catalog_manager.cc:5838] T 70fea649583648ca86b4b3673ab0305f P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.795941 17610 catalog_manager.cc:5838] T 0b99ff1920274831b58d8283bd592818 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.797752 19873 tablet_bootstrap.cc:654] T 66599c38b1a64ad28fe7bd8e5b5056a8 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:11.797744 17610 catalog_manager.cc:5838] T 924bc897f48b44069f1fb2ee71e26db7 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
I20260924 19:58:11.799721 17610 catalog_manager.cc:5838] T 274f3decdc7e42c0917bf28151c4da71 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 1 to 2, leader changed from eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131) to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 2 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
I20260924 19:58:11.801051 17610 catalog_manager.cc:5838] T bf025253c49a4c23b79a25333cad160e P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.803071 17610 catalog_manager.cc:5838] T f169c47d711d4566b2004cdfd0de725d P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.804322 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet f169c47d711d4566b2004cdfd0de725d (table default.Table_71fe66684fd042fb9d84f22528532866 [id=8d4e859f129f43b2a9a8e12861b6b803]). Expected version 3 got 0
I20260924 19:58:11.805053 17610 catalog_manager.cc:5838] T bea1ebdf647e4ef9a131a0398672b8ab P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } }
I20260924 19:58:11.806541 17610 catalog_manager.cc:5838] T 10fe8d2d06dd4f83bef8c52a82693344 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
I20260924 19:58:11.807832 17610 catalog_manager.cc:5838] T ca37ddd22f5d4bcab2862553408447ce P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
I20260924 19:58:11.809068 17610 catalog_manager.cc:5838] T bafaab10ce564b1b97659eccc0ab0967 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.811012 17610 catalog_manager.cc:5838] T e1137318a9fa4a0781f718dc34c4ecbe P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.812184 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet e1137318a9fa4a0781f718dc34c4ecbe (table default.Table_b2cb17472c5b450c93e06db475f000a0 [id=b4e755ee98cd40388e3681825580d7e8]). Expected version 1 got 0
I20260924 19:58:11.812901 17610 catalog_manager.cc:5838] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.814342 17610 catalog_manager.cc:5838] T 96379e7356ab40218c6ee2c126c0b4ae P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
W20260924 19:58:11.815099 18051 consensus_peers.cc:597] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:11.815955 17610 catalog_manager.cc:5838] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.817325 17610 catalog_manager.cc:5838] T 2a1ac7e1b192465696640c02898bc39f P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } }
I20260924 19:58:11.818557 17610 catalog_manager.cc:5838] T 7ba365c977984e68a316356465b1e36a P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.820001 17610 catalog_manager.cc:5838] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.821136 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet d930e7f4a6b7471899dab1b28e40f93d (table default.Table_9b8f6ad547fc4c3fa379c93590aebd56 [id=4fc8e565a6a640549716cee596fa3f9d]). Expected version 1 got 0
I20260924 19:58:11.821930 17610 catalog_manager.cc:5838] T 431d298f513d47268284d50af2ea1436 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.823455 17610 catalog_manager.cc:5838] T 18bb8f3d43df4aae8614b6dba9ffc668 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
W20260924 19:58:11.824257 18051 consensus_peers.cc:597] T 2e6d926ff7c24901a15b0c7b2239fbad P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:11.824857 17610 catalog_manager.cc:5838] T 3d25793b65da4315b6f95726374ea46c P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
I20260924 19:58:11.826025 17610 catalog_manager.cc:5838] T 6fd0d25961d74b67a42323a045f06b61 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
I20260924 19:58:11.827327 17610 catalog_manager.cc:5838] T 5e9f9b76a58e47c0beee5ce94a44b707 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.828711 17610 catalog_manager.cc:5838] T b25dd98712d54cd5b53df00b2cf9f3c4 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
I20260924 19:58:11.831112 17610 catalog_manager.cc:5838] T 7c1afbfc7de7480ab4be40680a3e3ada P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.832307 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet 7c1afbfc7de7480ab4be40680a3e3ada (table default.Table_a22d035e711649318fbcf76ad552a553 [id=31e1c3788d264823a12ecd9755ec2027]). Expected version 5 got 0
I20260924 19:58:11.832034 19873 tablet_bootstrap.cc:492] T 66599c38b1a64ad28fe7bd8e5b5056a8 P eaf64bfb64aa42948c9bc1411ce4ce35: No bootstrap required, opened a new log
I20260924 19:58:11.832998 17610 catalog_manager.cc:5838] T 8c05ec2ba01448d998c6ffa1525de984 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "c4f6e3a7b06148918d230c77ab0c8202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
I20260924 19:58:11.833956 19873 ts_tablet_manager.cc:1403] T 66599c38b1a64ad28fe7bd8e5b5056a8 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.001s
I20260924 19:58:11.834491 17610 catalog_manager.cc:5838] T d129e0979a584f40ba9a975349cf15a5 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.835744 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet d129e0979a584f40ba9a975349cf15a5 (table default.Table_a12f8684de2041ec83f2fee77829afdd [id=94cea8de30ca4010a95f142ac8408653]). Expected version 1 got 0
I20260924 19:58:11.836457 17610 catalog_manager.cc:5838] T 144355b18c2f413baa28dd632623db52 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } }
I20260924 19:58:11.836783 19873 raft_consensus.cc:359] T 66599c38b1a64ad28fe7bd8e5b5056a8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:58:11.837425 19873 raft_consensus.cc:385] T 66599c38b1a64ad28fe7bd8e5b5056a8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:11.837697 19873 raft_consensus.cc:740] T 66599c38b1a64ad28fe7bd8e5b5056a8 P eaf64bfb64aa42948c9bc1411ce4ce35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaf64bfb64aa42948c9bc1411ce4ce35, State: Initialized, Role: FOLLOWER
I20260924 19:58:11.837918 17610 catalog_manager.cc:5838] T b7a1887d396f40a1a0c153dd0b2fe307 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.839475 17610 catalog_manager.cc:5838] T 57102b1727e94a90981509c82c082441 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.841009 17610 catalog_manager.cc:5838] T 89646893588340038f30ed4754623ec2 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.842478 17610 catalog_manager.cc:5838] T 335f979eb04c49bcb53023693f54ad41 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.844163 17610 catalog_manager.cc:5838] T fbd5915ee98a4eda94479e9330cf8d04 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.845737 19873 consensus_queue.cc:260] T 66599c38b1a64ad28fe7bd8e5b5056a8 P eaf64bfb64aa42948c9bc1411ce4ce35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } }
I20260924 19:58:11.845562 17610 catalog_manager.cc:5838] T 7635c52883ec4434bad0cc15f8733f72 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.847074 17610 catalog_manager.cc:5838] T 97bbb17c170d42349d2c9cdb41349cb9 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.848443 17610 catalog_manager.cc:5838] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.849848 17610 catalog_manager.cc:5838] T 2de3877b94b44c94b6d0fdad93f8aa79 P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74e6ed351284d7fb110e44336ce643a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "f74e6ed351284d7fb110e44336ce643a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } } peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } } }
W20260924 19:58:11.851121 18051 consensus_peers.cc:597] T 92c0daa7d67e4aa390e334c669adda76 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:11.851720 19873 ts_tablet_manager.cc:1434] T 66599c38b1a64ad28fe7bd8e5b5056a8 P eaf64bfb64aa42948c9bc1411ce4ce35: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20260924 19:58:11.851123 17610 catalog_manager.cc:5838] T a00e3626ea154dc5a959f0fbb4e849dc P eaf64bfb64aa42948c9bc1411ce4ce35 reported cstate change: term changed from 0 to 1, leader changed from <none> to eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131). New cstate: current_term: 1 leader_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f6e3a7b06148918d230c77ab0c8202" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 42465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f74e6ed351284d7fb110e44336ce643a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 41301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaf64bfb64aa42948c9bc1411ce4ce35" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 40575 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:11.852439 17610 catalog_manager.cc:5938] TS eaf64bfb64aa42948c9bc1411ce4ce35 (127.17.2.131:40575) does not have the latest schema for tablet a00e3626ea154dc5a959f0fbb4e849dc (table default.Table_f1483a8ac50b4d38b827f0bbbc01dcc4 [id=522c4df3169f4d2ba1cbe15df69d40b3]). Expected version 2 got 0
W20260924 19:58:11.861575 18050 connection.cc:573] client connection to 127.17.2.130:41301 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:58:11.861703 18903 connection.cc:573] server connection from 127.17.2.130:55157 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:58:11.882264 19873 tablet_bootstrap.cc:492] T 56295c0109ce489fb03d65674df62c44 P eaf64bfb64aa42948c9bc1411ce4ce35: Bootstrap starting.
W20260924 19:58:11.894410 18053 connection.cc:573] server connection from 127.17.2.130:37897 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:58:11.899631 19880 negotiation.cc:336] Failed RPC negotiation. Trace:
0924 19:58:11.880852 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.17.2.130:41301 (local address 127.17.2.131:51655)
0924 19:58:11.888364 (+ 7512us) negotiation.cc:107] Waiting for socket to connect
0924 19:58:11.888405 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0924 19:58:11.888855 (+ 450us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0924 19:58:11.898506 (+ 9651us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.2.130:41301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":6937}
I20260924 19:58:11.905179 19873 tablet_bootstrap.cc:654] T 56295c0109ce489fb03d65674df62c44 P eaf64bfb64aa42948c9bc1411ce4ce35: Neither blocks nor log segments found. Creating new log.
W20260924 19:58:11.905372 18051 consensus_peers.cc:597] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.908566 18051 consensus_peers.cc:597] T d930e7f4a6b7471899dab1b28e40f93d P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer c4f6e3a7b06148918d230c77ab0c8202 (127.17.2.129:42465): Couldn't send request to peer c4f6e3a7b06148918d230c77ab0c8202. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.129:42465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.924645 18050 consensus_peers.cc:597] T c1684aaf9d984649bff8c4b0b12044b9 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.130:41301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.925073 18050 consensus_peers.cc:597] T 8523125e35554830aff04b17677d5ceb P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.130:41301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.925403 18050 consensus_peers.cc:597] T 5f116f866c2b4674b4eef531cf00af66 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.130:41301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.925716 18050 consensus_peers.cc:597] T edc6c16578e5463a9d44c25fd6c538a9 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.130:41301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.926041 18050 consensus_peers.cc:597] T 5df3dcf4ddcf4efa9aa047da56dfb7a7 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.130:41301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.926368 18050 consensus_peers.cc:597] T e59e737a68dd443f9898113b2b2a401e P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.130:41301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20260924 19:58:11.926677 18050 consensus_peers.cc:597] T a9e7c5b28fea4dbda802702c8b85ec54 P eaf64bfb64aa42948c9bc1411ce4ce35 -> Peer f74e6ed351284d7fb110e44336ce643a (127.17.2.130:41301): Couldn't send request to peer f74e6ed351284d7fb110e44336ce643a. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.130:41301: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:11.930573 17418 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu with pid 18027
I20260924 19:58:12.017017 17610 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_4e3a3158cde24302806b7ba7827831c4 [id=fa0b10df171340868b0305a28d192ea5] alter complete (version 1)
I20260924 19:58:12.037168 17610 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_dc1e895a797842afbf6054713d1c978d [id=738602c27af64fc2a1b0f1b1ddae9e2f] alter complete (version 1)
I20260924 19:58:12.063421 17610 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_7c27cdce0a3b40f69142de52aaf39493 [id=65f397bcbc164d0caa6cecae1bab74da] alter complete (version 1)
I20260924 19:58:12.081729 17610 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_88352246addd43f4be871c983413b2de [id=fa469a44fb3741d39036337129cc36a1] alter complete (version 2)
I20260924 19:58:12.101979 17610 catalog_manager.cc:6391] T 00000000000000000000000000000000 P e1ff67c5d57d474a9d92cc4c6de1287e: default.Table_88b623b6a00845afb22e24e34604e9ea [id=72c7d8cc98ee49679d8c7589a9ac0c8f] alter complete (version 2)
W20260924 19:58:12.120685 17520 connection.cc:573] server connection from 127.17.2.131:46743 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:58:12.123736 17418 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu with pid 18320
W20260924 19:58:12.138361 17601 connection.cc:466] server connection from 127.17.2.131:47219 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.17.2.131:47219 (request call id 122) could send its response
I20260924 19:58:12.183406 17418 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu with pid 17497
I20260924 19:58:12.243881 17418 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu with pid 17572
2026-09-24T19:58:12Z chronyd exiting
I20260924 19:58:12.323728 17418 master-stress-test.cc:587] Tables created: 137
I20260924 19:58:12.323943 17418 master-stress-test.cc:588] Tables altered: 51
I20260924 19:58:12.324136 17418 master-stress-test.cc:589] Tables deleted: 52
I20260924 19:58:12.324280 17418 master-stress-test.cc:590] Tablets replaced: 25
I20260924 19:58:12.324461 17418 master-stress-test.cc:591] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (65689 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
2026-09-24T19:58:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:58:12Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task0eKU8U/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task0eKU8U/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/security.properties
2026-09-24 19:58:20: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task0eKU8U/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task0eKU8U/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260924 19:58:29.211869 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.190:44951
--webserver_interface=127.17.2.190
--webserver_port=0
--builtin_ntp_servers=127.17.2.148:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--hive_metastore_uris=thrift://127.0.0.1:33597
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260924 19:58:29.530856 20177 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:58:29.531534 20177 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:58:29.532009 20177 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260924 19:58:29.565476 20177 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260924 19:58:29.565788 20177 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260924 19:58:29.566097 20177 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:58:29.566375 20177 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:58:29.566612 20177 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:58:29.566901 20177 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:58:29.605536 20177 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:33597
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.190:44951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.2.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:58:29.607003 20177 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:58:29.608740 20177 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260924 19:58:29.621186 20183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:29.621825 20184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:30.889351 20186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:30.892534 20185 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1269 milliseconds
W20260924 19:58:30.893657 20177 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.273s user 0.410s sys 0.852s
W20260924 19:58:30.893923 20177 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.273s user 0.410s sys 0.852s
I20260924 19:58:30.894202 20177 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260924 19:58:30.895494 20177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:58:30.899518 20177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:58:30.900987 20177 hybrid_clock.cc:648] HybridClock initialized: now 1790279910900949 us; error 53 us; skew 500 ppm
I20260924 19:58:30.901889 20177 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:58:30.913321 20177 webserver.cc:533] Webserver started at http://127.17.2.190:37257/ using document root <none> and password file <none>
I20260924 19:58:30.915262 20177 fs_manager.cc:362] Metadata directory not provided
I20260924 19:58:30.915618 20177 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:58:30.916352 20177 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:58:30.925292 20177 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/data/instance:
uuid: "1feab783ba9d4112932c72545e55ea19"
format_stamp: "Formatted at 2026-09-24 19:58:30 on dist-test-slave-h376"
I20260924 19:58:30.927132 20177 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/wal/instance:
uuid: "1feab783ba9d4112932c72545e55ea19"
format_stamp: "Formatted at 2026-09-24 19:58:30 on dist-test-slave-h376"
I20260924 19:58:30.940227 20177 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.005s sys 0.006s
I20260924 19:58:30.950419 20193 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:30.952250 20177 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.002s
I20260924 19:58:30.952783 20177 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/wal
uuid: "1feab783ba9d4112932c72545e55ea19"
format_stamp: "Formatted at 2026-09-24 19:58:30 on dist-test-slave-h376"
I20260924 19:58:30.953353 20177 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:58:31.052122 20177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:58:31.053678 20177 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:58:31.054154 20177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:58:31.133072 20177 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.190:44951
I20260924 19:58:31.133164 20244 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.190:44951 every 8 connection(s)
I20260924 19:58:31.136057 20177 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/data/info.pb
I20260924 19:58:31.141882 20245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:31.144315 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 20177
I20260924 19:58:31.144767 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-0/wal/instance
I20260924 19:58:31.147245 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.189:40117
--webserver_interface=127.17.2.189
--webserver_port=0
--builtin_ntp_servers=127.17.2.148:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--hive_metastore_uris=thrift://127.0.0.1:33597
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260924 19:58:31.152704 20245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:58:31.185076 20245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } has no permanent_uuid. Determining permanent_uuid...
W20260924 19:58:31.187963 20196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.189:40117: connect: Connection refused (error 111)
W20260924 19:58:31.192701 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:40117: Network error: Client connection negotiation failed: client connection to 127.17.2.189:40117: connect: Connection refused (error 111)
I20260924 19:58:31.242260 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } attempt: 1
W20260924 19:58:31.247690 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:40117: Network error: Client connection negotiation failed: client connection to 127.17.2.189:40117: connect: Connection refused (error 111)
I20260924 19:58:31.316181 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } attempt: 2
W20260924 19:58:31.320709 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:40117: Network error: Client connection negotiation failed: client connection to 127.17.2.189:40117: connect: Connection refused (error 111)
I20260924 19:58:31.412184 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } attempt: 3
W20260924 19:58:31.416532 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:40117: Network error: Client connection negotiation failed: client connection to 127.17.2.189:40117: connect: Connection refused (error 111)
W20260924 19:58:31.474130 20246 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:58:31.474799 20246 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:58:31.475342 20246 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260924 19:58:31.509974 20246 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260924 19:58:31.510317 20246 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260924 19:58:31.510591 20246 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:58:31.510854 20246 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:58:31.511113 20246 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:58:31.511439 20246 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:58:31.549979 20246 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:33597
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.189:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.2.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:58:31.551559 20246 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:58:31.553277 20246 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260924 19:58:31.559909 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } attempt: 4
W20260924 19:58:31.565877 20255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:31.566358 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:40117: Network error: Client connection negotiation failed: client connection to 127.17.2.189:40117: connect: Connection refused (error 111)
I20260924 19:58:31.865820 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } attempt: 5
W20260924 19:58:31.873132 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:40117: Network error: Client connection negotiation failed: client connection to 127.17.2.189:40117: connect: Connection refused (error 111)
I20260924 19:58:32.420779 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } attempt: 6
W20260924 19:58:32.436297 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.189:40117: Network error: Client connection negotiation failed: client connection to 127.17.2.189:40117: connect: Connection refused (error 111)
W20260924 19:58:31.567145 20256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:32.778872 20257 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20260924 19:58:32.781209 20258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:32.783984 20246 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.219s user 0.464s sys 0.754s
W20260924 19:58:32.784340 20246 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.219s user 0.464s sys 0.754s
I20260924 19:58:32.784650 20246 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260924 19:58:32.786371 20246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:58:32.789844 20246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:58:32.791378 20246 hybrid_clock.cc:648] HybridClock initialized: now 1790279912791325 us; error 40 us; skew 500 ppm
I20260924 19:58:32.792632 20246 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:58:32.807031 20246 webserver.cc:533] Webserver started at http://127.17.2.189:38345/ using document root <none> and password file <none>
I20260924 19:58:32.808451 20246 fs_manager.cc:362] Metadata directory not provided
I20260924 19:58:32.808759 20246 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:58:32.809396 20246 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:58:32.816860 20246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/data/instance:
uuid: "f09f90f3506b48cd9c797dbbeffd0721"
format_stamp: "Formatted at 2026-09-24 19:58:32 on dist-test-slave-h376"
I20260924 19:58:32.818746 20246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/wal/instance:
uuid: "f09f90f3506b48cd9c797dbbeffd0721"
format_stamp: "Formatted at 2026-09-24 19:58:32 on dist-test-slave-h376"
I20260924 19:58:32.830204 20246 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s
I20260924 19:58:32.838878 20267 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:32.840408 20246 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 19:58:32.840873 20246 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/wal
uuid: "f09f90f3506b48cd9c797dbbeffd0721"
format_stamp: "Formatted at 2026-09-24 19:58:32 on dist-test-slave-h376"
I20260924 19:58:32.841364 20246 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:58:32.941533 20246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:58:32.943137 20246 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:58:32.943614 20246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:58:33.017584 20246 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.189:40117
I20260924 19:58:33.017671 20318 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.189:40117 every 8 connection(s)
I20260924 19:58:33.020479 20246 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/data/info.pb
I20260924 19:58:33.025168 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 20246
I20260924 19:58:33.025606 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-1/wal/instance
I20260924 19:58:33.026378 20319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:33.028348 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.188:35311
--webserver_interface=127.17.2.188
--webserver_port=0
--builtin_ntp_servers=127.17.2.148:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--hive_metastore_uris=thrift://127.0.0.1:33597
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260924 19:58:33.037102 20319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:58:33.058285 20319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:58:33.073365 20319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } has no permanent_uuid. Determining permanent_uuid...
W20260924 19:58:33.075712 20270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
W20260924 19:58:33.079897 20319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:33.129415 20319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 1
W20260924 19:58:33.135699 20319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:33.204160 20319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 2
W20260924 19:58:33.209126 20319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:33.300616 20319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 3
W20260924 19:58:33.305208 20319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
W20260924 19:58:33.378118 20320 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:58:33.378717 20320 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:58:33.379112 20320 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260924 19:58:33.413432 20320 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260924 19:58:33.413679 20320 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260924 19:58:33.413889 20320 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:58:33.414089 20320 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:58:33.414390 20320 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:58:33.414613 20320 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:58:33.448671 20319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 4
W20260924 19:58:33.453157 20319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:33.453750 20320 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33597
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.188:35311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.2.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:58:33.455068 20320 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:58:33.456784 20320 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260924 19:58:33.468717 20330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:58:33.496881 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } attempt: 7
I20260924 19:58:33.515508 20245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } has no permanent_uuid. Determining permanent_uuid...
W20260924 19:58:33.525242 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:33.583820 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 1
W20260924 19:58:33.590978 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:33.672583 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 2
W20260924 19:58:33.679679 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:33.752648 20319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 5
W20260924 19:58:33.759768 20319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:33.765326 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 3
W20260924 19:58:33.772217 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:33.912863 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 4
W20260924 19:58:33.920104 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:34.203750 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 5
W20260924 19:58:34.211374 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:34.307310 20319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 6
W20260924 19:58:34.319531 20319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
W20260924 19:58:33.469292 20331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:34.718114 20332 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20260924 19:58:34.720126 20333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:34.722716 20320 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.254s user 0.472s sys 0.779s
W20260924 19:58:34.723013 20320 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.255s user 0.472s sys 0.779s
I20260924 19:58:34.723282 20320 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260924 19:58:34.724396 20320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:58:34.727048 20320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:58:34.728464 20320 hybrid_clock.cc:648] HybridClock initialized: now 1790279914728428 us; error 44 us; skew 500 ppm
I20260924 19:58:34.729308 20320 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:58:34.737165 20320 webserver.cc:533] Webserver started at http://127.17.2.188:36753/ using document root <none> and password file <none>
I20260924 19:58:34.738248 20320 fs_manager.cc:362] Metadata directory not provided
I20260924 19:58:34.738505 20320 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:58:34.738988 20320 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:58:34.743520 20320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data/instance:
uuid: "022c1898fcbe4856aa680c26047f6934"
format_stamp: "Formatted at 2026-09-24 19:58:34 on dist-test-slave-h376"
I20260924 19:58:34.744658 20320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal/instance:
uuid: "022c1898fcbe4856aa680c26047f6934"
format_stamp: "Formatted at 2026-09-24 19:58:34 on dist-test-slave-h376"
I20260924 19:58:34.753166 20320 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.007s
I20260924 19:58:34.759629 20344 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:34.760921 20320 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:58:34.761296 20320 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal
uuid: "022c1898fcbe4856aa680c26047f6934"
format_stamp: "Formatted at 2026-09-24 19:58:34 on dist-test-slave-h376"
I20260924 19:58:34.761672 20320 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:58:34.763935 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 6
W20260924 19:58:34.771931 20245 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.17.2.188:35311: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:34.842144 20320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:58:34.843832 20320 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:58:34.844295 20320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:58:34.916888 20320 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.188:35311
I20260924 19:58:34.916968 20397 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.188:35311 every 8 connection(s)
I20260924 19:58:34.919683 20320 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data/info.pb
I20260924 19:58:34.924916 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 20320
I20260924 19:58:34.925318 20398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:34.925354 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal/instance
I20260924 19:58:34.942847 20398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:58:34.969738 20398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:58:34.980796 20398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:58:34.996667 20398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: Bootstrap starting.
I20260924 19:58:35.002909 20398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:35.005064 20398 log.cc:826] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: Log is configured to *not* fsync() on all Append() calls
I20260924 19:58:35.010191 20398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: No bootstrap required, opened a new log
I20260924 19:58:35.036697 20398 raft_consensus.cc:359] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:35.037643 20398 raft_consensus.cc:385] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:35.038002 20398 raft_consensus.cc:740] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022c1898fcbe4856aa680c26047f6934, State: Initialized, Role: FOLLOWER
I20260924 19:58:35.039057 20398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:35.041935 20405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:35.042718 20405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:35.044097 20398 sys_catalog.cc:565] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:58:35.077469 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.077759 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.092490 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.092959 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.099552 20352 tablet.cc:2433] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:58:35.070215 20320 master_runner.cc:187] Error getting master registration for 127.17.2.190:44951: OK, instance_id { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" instance_seqno: 1790279911112774 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:58:35.102404 20320 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260924 19:58:35.109660 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.110255 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.127187 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.129320 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.145486 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.154407 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.169122 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.169574 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.182562 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.182809 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.196288 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.196568 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.209795 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.210078 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.224409 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.224658 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.237869 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.238121 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.251027 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.251366 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.264314 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.264535 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.277420 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.277655 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.290709 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.290972 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.304744 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.304986 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.318148 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.318396 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.331411 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.331656 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.344851 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.345134 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.358837 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.359067 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.372264 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.372520 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.380115 20319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 7
W20260924 19:58:35.389741 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.390017 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.403038 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.403385 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.404313 20319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Bootstrap starting.
I20260924 19:58:35.412045 20319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:35.414456 20319 log.cc:826] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Log is configured to *not* fsync() on all Append() calls
W20260924 19:58:35.418048 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.418298 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.420338 20319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: No bootstrap required, opened a new log
W20260924 19:58:35.431376 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.431641 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.447685 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.448100 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.448596 20319 raft_consensus.cc:359] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:35.449471 20319 raft_consensus.cc:385] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:35.449769 20319 raft_consensus.cc:740] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09f90f3506b48cd9c797dbbeffd0721, State: Initialized, Role: FOLLOWER
I20260924 19:58:35.450892 20319 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:35.453472 20425 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:35.454176 20425 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:35.455642 20319 sys_catalog.cc:565] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:58:35.462930 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.463408 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.477578 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.477921 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.491792 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.492234 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.507504 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.508037 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.513017 20405 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:35.513595 20405 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:35.520006 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022c1898fcbe4856aa680c26047f6934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:35.520916 20294 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022c1898fcbe4856aa680c26047f6934 in term 0.
W20260924 19:58:35.522657 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.522933 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.523862 20349 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934, f09f90f3506b48cd9c797dbbeffd0721; no voters:
I20260924 19:58:35.489125 20246 master_runner.cc:187] Error getting master registration for 127.17.2.190:44951: OK, instance_id { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" instance_seqno: 1790279911112774 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:58:35.525502 20246 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:58:35.527064 20405 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
I20260924 19:58:35.527822 20405 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:35.528463 20405 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:35.529101 20405 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:58:35.540369 20440 catalog_manager.cc:1607] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.540723 20440 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.543757 20405 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:35.546526 20405 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 1 election: Requested vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
I20260924 19:58:35.548118 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022c1898fcbe4856aa680c26047f6934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721"
I20260924 19:58:35.548667 20294 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:58:35.551065 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.551394 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.557075 20294 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022c1898fcbe4856aa680c26047f6934 in term 1.
W20260924 19:58:35.564227 20440 catalog_manager.cc:1607] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 19:58:35.565362 20349 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934, f09f90f3506b48cd9c797dbbeffd0721; no voters:
I20260924 19:58:35.566279 20405 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:35.566735 20405 raft_consensus.cc:697] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 1 LEADER]: Becoming Leader. State: Replica: 022c1898fcbe4856aa680c26047f6934, State: Running, Role: LEADER
I20260924 19:58:35.567715 20405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
W20260924 19:58:35.572613 20440 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.576444 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.577804 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.581836 20442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 022c1898fcbe4856aa680c26047f6934. Latest consensus state: current_term: 1 leader_uuid: "022c1898fcbe4856aa680c26047f6934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:35.582551 20442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: This master's current role is: LEADER
W20260924 19:58:35.591805 20419 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.592092 20419 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.592842 20440 catalog_manager.cc:1607] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.593138 20440 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.603395 20444 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260924 19:58:35.611343 20444 catalog_manager.cc:1533] Initializing Kudu cluster ID...
W20260924 19:58:35.613107 20440 catalog_manager.cc:1607] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.613933 20440 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:58:35.630205 20440 catalog_manager.cc:1607] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.631088 20440 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.643105 20294 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Refusing update from remote peer 022c1898fcbe4856aa680c26047f6934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 19:58:35.650784 20440 catalog_manager.cc:1607] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 19:58:35.651635 20442 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:58:35.651338 20440 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.669716 20425 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 022c1898fcbe4856aa680c26047f6934. Latest consensus state: current_term: 1 leader_uuid: "022c1898fcbe4856aa680c26047f6934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:35.670655 20425 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:58:35.673674 20440 catalog_manager.cc:1607] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:58:35.674031 20440 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.681766 20445 mvcc.cc:204] Tried to move back new op lower bound from 7332986534448324608 to 7332986534176944128. Current Snapshot: MvccSnapshot[applied={T|T < 7332986534448324608}]
I20260924 19:58:35.686419 20405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "022c1898fcbe4856aa680c26047f6934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:35.687249 20405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: This master's current role is: LEADER
I20260924 19:58:35.692521 20444 catalog_manager.cc:1396] Generated new cluster ID: 543042799ec94022838cc92e16f8796f
I20260924 19:58:35.692852 20444 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260924 19:58:35.720778 20425 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "022c1898fcbe4856aa680c26047f6934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:35.721566 20425 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:35.730592 20440 catalog_manager.cc:1308] Loaded cluster ID: 543042799ec94022838cc92e16f8796f
I20260924 19:58:35.730896 20440 catalog_manager.cc:1601] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: loading cluster ID for follower catalog manager: success
W20260924 19:58:35.735714 20440 catalog_manager.cc:1625] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260924 19:58:35.736074 20440 catalog_manager.cc:896] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.749678 20444 catalog_manager.cc:1419] Generated new certificate authority record
I20260924 19:58:35.751921 20444 catalog_manager.cc:1553] Loading token signing keys...
W20260924 19:58:35.769053 20440 catalog_manager.cc:1625] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260924 19:58:35.769902 20440 catalog_manager.cc:896] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:58:35.785526 20444 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: Generated new TSK 0
I20260924 19:58:35.789057 20444 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260924 19:58:35.799332 20444 catalog_manager.cc:1576] Loading latest processed Hive Metastore notification log event ID...
I20260924 19:58:35.804128 20440 catalog_manager.cc:1623] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: acquiring CA information for follower catalog manager: success
I20260924 19:58:35.805368 20245 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } attempt: 7
I20260924 19:58:35.814430 20444 catalog_manager.cc:6068] Last processed Hive Metastore notification event ID: -1
I20260924 19:58:35.831507 20440 catalog_manager.cc:1651] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:58:35.844856 20245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19: Bootstrap starting.
I20260924 19:58:35.852991 20245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:35.856511 20245 log.cc:826] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19: Log is configured to *not* fsync() on all Append() calls
I20260924 19:58:35.865389 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.2.129:0
--local_ip_for_outbound_sockets=127.17.2.129
--webserver_interface=127.17.2.129
--webserver_port=0
--tserver_master_addrs=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--builtin_ntp_servers=127.17.2.148:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260924 19:58:35.873119 20245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19: No bootstrap required, opened a new log
I20260924 19:58:35.907454 20245 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:35.909461 20245 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:35.909858 20245 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1feab783ba9d4112932c72545e55ea19, State: Initialized, Role: FOLLOWER
I20260924 19:58:35.911073 20245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:35.936312 20245 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:58:35.936866 20218 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:35.939980 20219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022c1898fcbe4856aa680c26047f6934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19"
I20260924 19:58:35.939635 20220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022c1898fcbe4856aa680c26047f6934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:35.957510 20218 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Refusing update from remote peer 022c1898fcbe4856aa680c26047f6934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:58:35.958599 20454 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:35.959323 20454 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:35.960635 20453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:35.961557 20453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:35.964435 20454 leader_election.cc:290] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:35.966137 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:35.967770 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:35.970832 20196 leader_election.cc:304] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1feab783ba9d4112932c72545e55ea19; no voters: 022c1898fcbe4856aa680c26047f6934, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:35.972446 20405 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:35.977553 20454 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:36.075119 20454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 022c1898fcbe4856aa680c26047f6934. Latest consensus state: current_term: 1 leader_uuid: "022c1898fcbe4856aa680c26047f6934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:36.075860 20454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:36.076993 20453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "022c1898fcbe4856aa680c26047f6934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:36.077662 20453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:36.185564 20473 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:58:36.223585 20473 catalog_manager.cc:1308] Loaded cluster ID: 543042799ec94022838cc92e16f8796f
I20260924 19:58:36.223960 20473 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19: loading cluster ID for follower catalog manager: success
I20260924 19:58:36.242702 20473 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19: acquiring CA information for follower catalog manager: success
I20260924 19:58:36.252628 20473 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:58:36.312652 20475 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:36.313498 20475 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:36.323974 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:36.325459 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:36.328598 20268 leader_election.cc:304] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09f90f3506b48cd9c797dbbeffd0721; no voters: 022c1898fcbe4856aa680c26047f6934, 1feab783ba9d4112932c72545e55ea19
I20260924 19:58:36.331689 20475 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:36.333520 20475 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:36.443975 20477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "022c1898fcbe4856aa680c26047f6934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:36.446138 20477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: This master's current role is: LEADER
W20260924 19:58:36.696400 20452 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:58:36.697144 20452 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:58:36.698113 20452 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260924 19:58:36.764959 20452 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:58:36.766662 20452 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.2.129
I20260924 19:58:36.841488 20452 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.2.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.2.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.2.129
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:58:36.843272 20452 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:58:36.846081 20452 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260924 19:58:36.887341 20485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:58:36.908246 20486 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:36.909116 20486 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:36.912812 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:36.921890 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:36.923906 20270 leader_election.cc:304] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09f90f3506b48cd9c797dbbeffd0721; no voters: 022c1898fcbe4856aa680c26047f6934, 1feab783ba9d4112932c72545e55ea19
I20260924 19:58:36.926330 20486 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:36.926371 20489 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:36.967585 20490 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:36.972972 20490 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:36.982915 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:36.991062 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:36.993525 20196 leader_election.cc:304] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1feab783ba9d4112932c72545e55ea19; no voters: 022c1898fcbe4856aa680c26047f6934, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:36.995714 20490 leader_election.cc:290] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:36.997987 20490 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:37.995988 20495 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:37.996532 20495 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:37.998624 20495 leader_election.cc:290] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:38.000984 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:38.002847 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:38.007040 20196 leader_election.cc:304] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1feab783ba9d4112932c72545e55ea19; no voters: 022c1898fcbe4856aa680c26047f6934, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:38.008106 20495 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:38.594740 20496 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:38.595331 20496 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:38.597442 20496 leader_election.cc:290] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:38.598208 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:38.598634 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:38.600814 20196 leader_election.cc:304] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1feab783ba9d4112932c72545e55ea19; no voters: 022c1898fcbe4856aa680c26047f6934, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:38.603425 20496 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:39.266347 20497 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:39.273505 20497 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:39.276768 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:39.279659 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:39.281291 20270 leader_election.cc:304] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09f90f3506b48cd9c797dbbeffd0721; no voters: 022c1898fcbe4856aa680c26047f6934, 1feab783ba9d4112932c72545e55ea19
I20260924 19:58:39.287870 20497 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:39.288852 20497 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:58:38.281864 20484 debug-util.cc:398] Leaking SignalData structure 0x7b0800037240 after lost signal to thread 20452
W20260924 19:58:36.892434 20487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:39.385810 20452 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 2.496s user 0.797s sys 1.343s
W20260924 19:58:39.386380 20452 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 2.496s user 0.797s sys 1.343s
I20260924 19:58:39.395815 20452 server_base.cc:1061] running on GCE node
W20260924 19:58:39.396814 20500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:58:39.398144 20452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:58:39.400512 20452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:58:39.401921 20452 hybrid_clock.cc:648] HybridClock initialized: now 1790279919401880 us; error 41 us; skew 500 ppm
I20260924 19:58:39.403302 20452 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:58:39.409842 20452 webserver.cc:533] Webserver started at http://127.17.2.129:41541/ using document root <none> and password file <none>
I20260924 19:58:39.410843 20452 fs_manager.cc:362] Metadata directory not provided
I20260924 19:58:39.411062 20452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:58:39.411597 20452 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:58:39.416591 20452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/data/instance:
uuid: "30a4f50fc28c4616ba622ba51b0b2622"
format_stamp: "Formatted at 2026-09-24 19:58:39 on dist-test-slave-h376"
I20260924 19:58:39.417730 20452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/instance:
uuid: "30a4f50fc28c4616ba622ba51b0b2622"
format_stamp: "Formatted at 2026-09-24 19:58:39 on dist-test-slave-h376"
I20260924 19:58:39.426025 20452 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260924 19:58:39.432446 20505 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:39.433697 20452 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260924 19:58:39.434032 20452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal
uuid: "30a4f50fc28c4616ba622ba51b0b2622"
format_stamp: "Formatted at 2026-09-24 19:58:39 on dist-test-slave-h376"
I20260924 19:58:39.434353 20452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:58:39.504424 20452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:58:39.506295 20452 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:58:39.507217 20452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:58:39.511101 20452 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260924 19:58:39.517781 20452 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:58:39.518102 20452 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:39.518430 20452 ts_tablet_manager.cc:616] Registered 0 tablets
I20260924 19:58:39.518666 20452 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:39.599294 20552 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:39.600517 20552 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:39.603669 20552 leader_election.cc:290] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:39.604949 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:39.605293 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:39.607266 20196 leader_election.cc:304] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1feab783ba9d4112932c72545e55ea19; no voters: 022c1898fcbe4856aa680c26047f6934, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:39.608531 20552 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:39.700249 20452 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.129:33667
I20260924 19:58:39.700392 20619 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.129:33667 every 8 connection(s)
I20260924 19:58:39.703987 20452 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/data/info.pb
I20260924 19:58:39.709174 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 20452
I20260924 19:58:39.709924 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/instance
I20260924 19:58:39.733229 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.2.130:0
--local_ip_for_outbound_sockets=127.17.2.130
--webserver_interface=127.17.2.130
--webserver_port=0
--tserver_master_addrs=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--builtin_ntp_servers=127.17.2.148:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260924 19:58:39.784051 20620 heartbeater.cc:344] Connected to a master server at 127.17.2.188:35311
I20260924 19:58:39.784564 20620 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:39.785952 20620 heartbeater.cc:507] Master 127.17.2.188:35311 requested a full tablet report, sending...
I20260924 19:58:39.786036 20622 heartbeater.cc:344] Connected to a master server at 127.17.2.190:44951
I20260924 19:58:39.786433 20622 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:39.787407 20622 heartbeater.cc:507] Master 127.17.2.190:44951 requested a full tablet report, sending...
I20260924 19:58:39.789924 20621 heartbeater.cc:344] Connected to a master server at 127.17.2.189:40117
I20260924 19:58:39.789983 20363 ts_manager.cc:194] Registered new tserver with Master: 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667)
I20260924 19:58:39.790304 20621 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:39.791289 20210 ts_manager.cc:194] Registered new tserver with Master: 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667)
I20260924 19:58:39.791751 20621 heartbeater.cc:507] Master 127.17.2.189:40117 requested a full tablet report, sending...
I20260924 19:58:39.792680 20363 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.17.2.129:42121
I20260924 19:58:39.794750 20284 ts_manager.cc:194] Registered new tserver with Master: 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667)
I20260924 19:58:39.809048 20631 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:39.809525 20631 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:39.811770 20631 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:39.812758 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:39.813357 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:39.814862 20270 leader_election.cc:304] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09f90f3506b48cd9c797dbbeffd0721; no voters: 022c1898fcbe4856aa680c26047f6934, 1feab783ba9d4112932c72545e55ea19
I20260924 19:58:39.815922 20631 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:58:40.146066 20630 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:58:40.146651 20630 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:58:40.147303 20630 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260924 19:58:40.152725 20636 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:40.153187 20636 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:40.155267 20636 leader_election.cc:290] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:40.156509 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:40.157778 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:40.159446 20196 leader_election.cc:304] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1feab783ba9d4112932c72545e55ea19; no voters: 022c1898fcbe4856aa680c26047f6934, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:40.160583 20636 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:58:40.186589 20630 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:58:40.187615 20630 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.2.130
I20260924 19:58:40.230453 20630 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.2.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.2.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.2.130
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:58:40.231864 20630 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:58:40.233477 20630 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260924 19:58:40.245751 20638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:58:40.298756 20620 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:40.398399 20644 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:40.399137 20644 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:40.403229 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:40.403628 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:40.406297 20270 leader_election.cc:304] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09f90f3506b48cd9c797dbbeffd0721; no voters: 022c1898fcbe4856aa680c26047f6934, 1feab783ba9d4112932c72545e55ea19
I20260924 19:58:40.408859 20644 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:40.409029 20645 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:40.943486 20646 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:40.944135 20646 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:40.948244 20646 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:40.948326 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:40.950933 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:40.954114 20270 leader_election.cc:304] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09f90f3506b48cd9c797dbbeffd0721; no voters: 022c1898fcbe4856aa680c26047f6934, 1feab783ba9d4112932c72545e55ea19
I20260924 19:58:40.955651 20646 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:41.490636 20648 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:41.491490 20648 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:41.495307 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:41.495633 20648 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:41.496398 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:41.498278 20270 leader_election.cc:304] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09f90f3506b48cd9c797dbbeffd0721; no voters: 022c1898fcbe4856aa680c26047f6934, 1feab783ba9d4112932c72545e55ea19
I20260924 19:58:41.499868 20648 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:58:40.247664 20639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:41.548688 20641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:41.550892 20640 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1300 milliseconds
I20260924 19:58:41.550987 20630 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260924 19:58:41.552263 20630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:58:41.555465 20630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:58:41.556927 20630 hybrid_clock.cc:648] HybridClock initialized: now 1790279921556880 us; error 58 us; skew 500 ppm
I20260924 19:58:41.557785 20630 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:58:41.564596 20630 webserver.cc:533] Webserver started at http://127.17.2.130:46079/ using document root <none> and password file <none>
I20260924 19:58:41.565559 20630 fs_manager.cc:362] Metadata directory not provided
I20260924 19:58:41.565806 20630 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:58:41.566280 20630 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:58:41.570889 20630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/data/instance:
uuid: "af20c71727a14e398a217d4623d1399a"
format_stamp: "Formatted at 2026-09-24 19:58:41 on dist-test-slave-h376"
I20260924 19:58:41.572170 20630 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/instance:
uuid: "af20c71727a14e398a217d4623d1399a"
format_stamp: "Formatted at 2026-09-24 19:58:41 on dist-test-slave-h376"
I20260924 19:58:41.580189 20630 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260924 19:58:41.586519 20653 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:41.587796 20630 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:58:41.588157 20630 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal
uuid: "af20c71727a14e398a217d4623d1399a"
format_stamp: "Formatted at 2026-09-24 19:58:41 on dist-test-slave-h376"
I20260924 19:58:41.588546 20630 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:58:41.619004 20654 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:41.619419 20654 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:41.621241 20654 leader_election.cc:290] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:41.622208 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:41.622476 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:41.623934 20196 leader_election.cc:304] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1feab783ba9d4112932c72545e55ea19; no voters: 022c1898fcbe4856aa680c26047f6934, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:41.624672 20654 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:41.650425 20630 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:58:41.652231 20630 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:58:41.652745 20630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:58:41.655773 20630 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260924 19:58:41.660923 20630 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:58:41.661258 20630 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:41.661564 20630 ts_tablet_manager.cc:616] Registered 0 tablets
I20260924 19:58:41.661723 20630 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:41.819933 20630 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.130:38719
I20260924 19:58:41.820216 20766 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.130:38719 every 8 connection(s)
I20260924 19:58:41.824066 20630 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/data/info.pb
I20260924 19:58:41.824756 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 20630
I20260924 19:58:41.825268 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/instance
I20260924 19:58:41.841053 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.2.131:0
--local_ip_for_outbound_sockets=127.17.2.131
--webserver_interface=127.17.2.131
--webserver_port=0
--tserver_master_addrs=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--builtin_ntp_servers=127.17.2.148:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260924 19:58:41.885730 20767 heartbeater.cc:344] Connected to a master server at 127.17.2.188:35311
I20260924 19:58:41.886252 20767 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:41.887691 20767 heartbeater.cc:507] Master 127.17.2.188:35311 requested a full tablet report, sending...
I20260924 19:58:41.891109 20363 ts_manager.cc:194] Registered new tserver with Master: af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:41.891371 20768 heartbeater.cc:344] Connected to a master server at 127.17.2.189:40117
I20260924 19:58:41.891651 20768 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:41.892557 20768 heartbeater.cc:507] Master 127.17.2.189:40117 requested a full tablet report, sending...
I20260924 19:58:41.892902 20363 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.17.2.130:59903
I20260924 19:58:41.894690 20771 heartbeater.cc:344] Connected to a master server at 127.17.2.190:44951
I20260924 19:58:41.895040 20771 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:41.895980 20771 heartbeater.cc:507] Master 127.17.2.190:44951 requested a full tablet report, sending...
I20260924 19:58:41.896217 20284 ts_manager.cc:194] Registered new tserver with Master: af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:41.899390 20210 ts_manager.cc:194] Registered new tserver with Master: af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 19:58:42.247279 20777 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:58:42.247776 20777 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:58:42.248288 20777 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260924 19:58:42.282833 20777 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:58:42.283823 20777 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.2.131
I20260924 19:58:42.322041 20777 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.2.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.2.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.2.131
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:58:42.323644 20777 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:58:42.325642 20777 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260924 19:58:42.340636 20783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:58:42.398228 20767 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:42.441078 20789 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:42.442624 20789 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:42.447928 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:42.451365 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:42.455183 20270 leader_election.cc:304] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09f90f3506b48cd9c797dbbeffd0721; no voters: 022c1898fcbe4856aa680c26047f6934, 1feab783ba9d4112932c72545e55ea19
I20260924 19:58:42.462325 20789 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:42.464234 20789 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:42.532785 20791 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:42.533806 20791 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:42.546221 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:42.553858 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:42.556535 20196 leader_election.cc:304] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1feab783ba9d4112932c72545e55ea19; no voters: 022c1898fcbe4856aa680c26047f6934, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:42.559222 20792 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:42.559589 20791 leader_election.cc:290] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
W20260924 19:58:43.749370 20782 debug-util.cc:398] Leaking SignalData structure 0x7b0800037240 after lost signal to thread 20777
W20260924 19:58:42.340991 20784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:43.894618 20777 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.554s user 0.536s sys 0.994s
W20260924 19:58:43.895325 20777 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.555s user 0.536s sys 0.994s
W20260924 19:58:43.897099 20786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:43.901582 20785 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1560 milliseconds
I20260924 19:58:43.901603 20777 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260924 19:58:43.902880 20777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:58:43.905586 20777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:58:43.907019 20777 hybrid_clock.cc:648] HybridClock initialized: now 1790279923906975 us; error 55 us; skew 500 ppm
I20260924 19:58:43.907881 20777 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:58:43.913862 20777 webserver.cc:533] Webserver started at http://127.17.2.131:39103/ using document root <none> and password file <none>
I20260924 19:58:43.914795 20777 fs_manager.cc:362] Metadata directory not provided
I20260924 19:58:43.915041 20777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:58:43.915653 20777 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:58:43.921766 20777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/data/instance:
uuid: "0055218dbcbd48b58018e9774e59e14c"
format_stamp: "Formatted at 2026-09-24 19:58:43 on dist-test-slave-h376"
I20260924 19:58:43.923446 20777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/instance:
uuid: "0055218dbcbd48b58018e9774e59e14c"
format_stamp: "Formatted at 2026-09-24 19:58:43 on dist-test-slave-h376"
I20260924 19:58:43.928303 20797 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:43.928694 20797 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:43.931535 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:43.932215 20373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:43.934805 20270 leader_election.cc:304] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f09f90f3506b48cd9c797dbbeffd0721; no voters: 022c1898fcbe4856aa680c26047f6934, 1feab783ba9d4112932c72545e55ea19
I20260924 19:58:43.934828 20797 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:43.935532 20797 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:43.936285 20777 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.005s sys 0.006s
I20260924 19:58:43.942541 20800 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:43.943913 20777 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:58:43.944335 20777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal
uuid: "0055218dbcbd48b58018e9774e59e14c"
format_stamp: "Formatted at 2026-09-24 19:58:43 on dist-test-slave-h376"
I20260924 19:58:43.944687 20777 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:58:44.002571 20777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:58:44.004720 20777 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:58:44.005396 20777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:58:44.008814 20777 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260924 19:58:44.013212 20777 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:58:44.013413 20777 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:44.013609 20777 ts_tablet_manager.cc:616] Registered 0 tablets
I20260924 19:58:44.013742 20777 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:44.188054 20777 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.131:41601
I20260924 19:58:44.188114 20912 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.131:41601 every 8 connection(s)
I20260924 19:58:44.192165 20777 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/data/info.pb
I20260924 19:58:44.195056 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 20777
I20260924 19:58:44.195631 17418 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/instance
I20260924 19:58:44.233476 20913 heartbeater.cc:344] Connected to a master server at 127.17.2.188:35311
I20260924 19:58:44.234021 20913 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:44.235345 20913 heartbeater.cc:507] Master 127.17.2.188:35311 requested a full tablet report, sending...
I20260924 19:58:44.238864 20362 ts_manager.cc:194] Registered new tserver with Master: 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:44.240972 20362 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.17.2.131:55075
I20260924 19:58:44.248008 20914 heartbeater.cc:344] Connected to a master server at 127.17.2.189:40117
I20260924 19:58:44.248497 20914 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:44.249217 20914 heartbeater.cc:507] Master 127.17.2.189:40117 requested a full tablet report, sending...
I20260924 19:58:44.247999 20915 heartbeater.cc:344] Connected to a master server at 127.17.2.190:44951
I20260924 19:58:44.251274 20915 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:44.251940 20915 heartbeater.cc:507] Master 127.17.2.190:44951 requested a full tablet report, sending...
I20260924 19:58:44.252094 20284 ts_manager.cc:194] Registered new tserver with Master: 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:44.255079 20210 ts_manager.cc:194] Registered new tserver with Master: 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:44.271207 17418 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260924 19:58:44.320446 20362 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50890:
name: "default.Table_11b41e77c9bf468c95a9cbe331cdf531"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:44.322392 20363 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50890:
name: "default.Table_d74dff5eb61a4b6fb10a13c1ec755df9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:44.322830 20361 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50890:
name: "default.Table_9d6179f80e96408995081e5524a3422a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:44.328205 20360 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50890:
name: "default.Table_f08030d9251649dab7ae5676c47c84bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:58:44.334317 20362 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_11b41e77c9bf468c95a9cbe331cdf531 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:44.355814 20363 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:44.356026 20360 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f08030d9251649dab7ae5676c47c84bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:44.360540 20361 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9d6179f80e96408995081e5524a3422a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:44.373347 17418 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu with pid 20320
W20260924 19:58:44.410251 20925 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.2.188:35311)
I20260924 19:58:44.418645 20947 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:44.419272 20947 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:44.422258 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:44.427763 20947 leader_election.cc:290] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
W20260924 19:58:44.430207 20196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20260924 19:58:44.441179 20196 leader_election.cc:336] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311): Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:44.441565 20196 leader_election.cc:304] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1feab783ba9d4112932c72545e55ea19; no voters: 022c1898fcbe4856aa680c26047f6934, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:44.442256 20947 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:58:44.460503 20767 heartbeater.cc:646] Failed to heartbeat to 127.17.2.188:35311 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:44.495280 20951 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:44.495687 20951 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:44.498211 20951 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:44.498983 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:44.499621 20218 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f09f90f3506b48cd9c797dbbeffd0721 in term 1.
I20260924 19:58:44.500772 20268 leader_election.cc:304] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721; no voters:
W20260924 19:58:44.501232 20270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260924 19:58:44.501502 20951 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:58:44.501863 20951 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 022c1898fcbe4856aa680c26047f6934)
I20260924 19:58:44.502328 20951 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:58:44.513688 20951 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:44.516819 20951 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 election: Requested vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:44.517601 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f09f90f3506b48cd9c797dbbeffd0721" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19"
I20260924 19:58:44.518164 20218 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 1 FOLLOWER]: Advancing to term 2
W20260924 19:58:44.523183 20270 leader_election.cc:336] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311): Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:44.525758 20218 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f09f90f3506b48cd9c797dbbeffd0721 in term 2.
W20260924 19:58:44.526376 20270 leader_election.cc:336] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311): Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:44.526942 20268 leader_election.cc:304] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721; no voters: 022c1898fcbe4856aa680c26047f6934
I20260924 19:58:44.527964 20951 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:58:44.540881 20951 raft_consensus.cc:697] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 2 LEADER]: Becoming Leader. State: Replica: f09f90f3506b48cd9c797dbbeffd0721, State: Running, Role: LEADER
I20260924 19:58:44.542026 20951 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:44.555478 20954 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09f90f3506b48cd9c797dbbeffd0721. Latest consensus state: current_term: 2 leader_uuid: "f09f90f3506b48cd9c797dbbeffd0721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:44.556447 20954 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: This master's current role is: LEADER
I20260924 19:58:44.559089 20956 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260924 19:58:44.574137 20956 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260924 19:58:44.578666 20956 catalog_manager.cc:1308] Loaded cluster ID: 543042799ec94022838cc92e16f8796f
I20260924 19:58:44.579037 20956 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260924 19:58:44.583846 20956 catalog_manager.cc:1553] Loading token signing keys...
I20260924 19:58:44.588253 20956 catalog_manager.cc:6230] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Loaded TSK: 0
I20260924 19:58:44.589697 20956 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260924 19:58:44.593834 20956 catalog_manager.cc:1576] Loading latest processed Hive Metastore notification log event ID...
I20260924 19:58:44.598057 20956 catalog_manager.cc:6068] Last processed Hive Metastore notification event ID: -1
I20260924 19:58:44.623204 20284 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_f08030d9251649dab7ae5676c47c84bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:44.626890 20283 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_9d6179f80e96408995081e5524a3422a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:44.627599 20282 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_d74dff5eb61a4b6fb10a13c1ec755df9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:44.629268 20281 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_11b41e77c9bf468c95a9cbe331cdf531"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:58:44.635737 20284 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f08030d9251649dab7ae5676c47c84bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:44.644482 20282 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:44.648205 20281 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_11b41e77c9bf468c95a9cbe331cdf531 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:44.653180 20283 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9d6179f80e96408995081e5524a3422a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:44.745045 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
W20260924 19:58:44.749360 20913 heartbeater.cc:646] Failed to heartbeat to 127.17.2.188:35311 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
W20260924 19:58:44.879962 20620 heartbeater.cc:646] Failed to heartbeat to 127.17.2.188:35311 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111)
I20260924 19:58:44.932478 17418 external_mini_cluster.cc:1372] Running /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
/tmp/dist-test-task0eKU8U/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.188:35311
--webserver_interface=127.17.2.188
--webserver_port=36753
--builtin_ntp_servers=127.17.2.148:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--hive_metastore_uris=thrift://127.0.0.1:33597
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260924 19:58:45.010907 20218 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 2 FOLLOWER]: Refusing update from remote peer f09f90f3506b48cd9c797dbbeffd0721: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260924 19:58:45.012915 20954 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
19:58:45.043 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_f08030d9251649dab7ae5676c47c84bc already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at jdk.proxy2/jdk.proxy2.$Proxy27.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
W20260924 19:58:45.049253 20284 catalog_manager.cc:2209] Already present: failed to create HMS catalog entry for table [id=5b4e3119e5284f8aa18a1b323d05b08a]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_f08030d9251649dab7ae5676c47c84bc already exists)
I20260924 19:58:45.062757 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09f90f3506b48cd9c797dbbeffd0721. Latest consensus state: current_term: 2 leader_uuid: "f09f90f3506b48cd9c797dbbeffd0721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:45.063494 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:45.071458 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f09f90f3506b48cd9c797dbbeffd0721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:45.072139 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:45.077221 20284 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_14cbcbeb7f98417197bb13b1f8b5af49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:45.079197 20954 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f09f90f3506b48cd9c797dbbeffd0721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:45.079841 20954 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: This master's current role is: LEADER
W20260924 19:58:45.110188 20270 consensus_peers.cc:597] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 -> Peer 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311): Couldn't send request to peer 022c1898fcbe4856aa680c26047f6934. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:45.178476 20281 catalog_manager.cc:2213] created HMS catalog entry for table [id=a5e0426bb5db466a960323a3d0d07a29]
I20260924 19:58:45.232707 20281 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_11b41e77c9bf468c95a9cbe331cdf531" } new_table_name: "default.Table_92546dd5153b4d58af6fc19d4dee39f0" modify_external_catalogs: true
I20260924 19:58:45.266110 20914 heartbeater.cc:499] Master 127.17.2.189:40117 was elected leader, sending a full tablet report...
I20260924 19:58:45.315639 20555 tablet_service.cc:1511] Processing CreateTablet for tablet 585daaec60a245308ac55d92aefae38f (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.318672 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585daaec60a245308ac55d92aefae38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.318221 20702 tablet_service.cc:1511] Processing CreateTablet for tablet 585daaec60a245308ac55d92aefae38f (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.318230 20848 tablet_service.cc:1511] Processing CreateTablet for tablet 585daaec60a245308ac55d92aefae38f (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.320252 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585daaec60a245308ac55d92aefae38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.320669 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585daaec60a245308ac55d92aefae38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.345238 20283 catalog_manager.cc:2213] created HMS catalog entry for table [id=966de0fe489d49ea9ae3954d0d0916c0]
I20260924 19:58:45.378798 20621 heartbeater.cc:499] Master 127.17.2.189:40117 was elected leader, sending a full tablet report...
I20260924 19:58:45.378060 20283 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_9d6179f80e96408995081e5524a3422a" } new_table_name: "default.Table_1ed870b0a11e446c86e6a4c6a15e81b6" modify_external_catalogs: true
I20260924 19:58:45.388789 20554 tablet_service.cc:1511] Processing CreateTablet for tablet eb59c659ddf1497182e2fe51c5c68c51 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.390025 20554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb59c659ddf1497182e2fe51c5c68c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.398159 20847 tablet_service.cc:1511] Processing CreateTablet for tablet eb59c659ddf1497182e2fe51c5c68c51 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.399721 20847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb59c659ddf1497182e2fe51c5c68c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.401435 20976 tablet_bootstrap.cc:492] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:45.404132 20701 tablet_service.cc:1511] Processing CreateTablet for tablet eb59c659ddf1497182e2fe51c5c68c51 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.405390 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb59c659ddf1497182e2fe51c5c68c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.415457 20976 tablet_bootstrap.cc:654] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:45.430429 20978 tablet_bootstrap.cc:492] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:45.433514 20280 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_08eddf6c85b94591a21958b5f18ada13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:45.433964 20977 tablet_bootstrap.cc:492] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:45.436753 20976 log.cc:826] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:58:45.443852 20279 master_service.cc:960] ReplaceTablet: received request to replace tablet eb59c659ddf1497182e2fe51c5c68c51 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:45.447815 20978 tablet_bootstrap.cc:654] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:45.449389 20278 master_service.cc:960] ReplaceTablet: received request to replace tablet 585daaec60a245308ac55d92aefae38f from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:45.460234 20978 log.cc:826] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:58:45.463447 20279 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet eb59c659ddf1497182e2fe51c5c68c51
I20260924 19:58:45.465694 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet eb59c659ddf1497182e2fe51c5c68c51 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c40348984314595b6e6e49c4f03f35e) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:45.467496 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet eb59c659ddf1497182e2fe51c5c68c51 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c40348984314595b6e6e49c4f03f35e) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:45.467875 20279 catalog_manager.cc:6934] ReplaceTablet: tablet eb59c659ddf1497182e2fe51c5c68c51 deleted and replaced by tablet 0c40348984314595b6e6e49c4f03f35e
I20260924 19:58:45.470064 20847 tablet_service.cc:1558] Processing DeleteTablet for tablet eb59c659ddf1497182e2fe51c5c68c51 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c40348984314595b6e6e49c4f03f35e) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:45.477406 20278 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 585daaec60a245308ac55d92aefae38f
I20260924 19:58:45.479075 20278 catalog_manager.cc:6934] ReplaceTablet: tablet 585daaec60a245308ac55d92aefae38f deleted and replaced by tablet 1592544a94794c81ab2f05d91ac67200
I20260924 19:58:45.480719 20978 tablet_bootstrap.cc:492] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:45.481271 20978 ts_tablet_manager.cc:1403] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.004s
I20260924 19:58:45.484630 20282 catalog_manager.cc:2213] created HMS catalog entry for table [id=fb6b0484c7d2481eb450f18797dac6b1]
I20260924 19:58:45.486547 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 585daaec60a245308ac55d92aefae38f with delete_type TABLET_DATA_DELETED (replaced by tablet 1592544a94794c81ab2f05d91ac67200) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:45.487663 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet 585daaec60a245308ac55d92aefae38f with delete_type TABLET_DATA_DELETED (replaced by tablet 1592544a94794c81ab2f05d91ac67200) from {username='slave'} at 127.0.0.1:60248
W20260924 19:58:45.490048 20271 catalog_manager.cc:5043] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet eb59c659ddf1497182e2fe51c5c68c51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb59c659ddf1497182e2fe51c5c68c51 already in progress: creating tablet
W20260924 19:58:45.490829 20271 catalog_manager.cc:5043] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet eb59c659ddf1497182e2fe51c5c68c51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb59c659ddf1497182e2fe51c5c68c51 already in progress: creating tablet
I20260924 19:58:45.491626 20768 heartbeater.cc:499] Master 127.17.2.189:40117 was elected leader, sending a full tablet report...
W20260924 19:58:45.491667 20271 catalog_manager.cc:5043] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet 585daaec60a245308ac55d92aefae38f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585daaec60a245308ac55d92aefae38f already in progress: creating tablet
I20260924 19:58:45.489362 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet 585daaec60a245308ac55d92aefae38f with delete_type TABLET_DATA_DELETED (replaced by tablet 1592544a94794c81ab2f05d91ac67200) from {username='slave'} at 127.0.0.1:48342
W20260924 19:58:45.502501 20271 catalog_manager.cc:5043] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet 585daaec60a245308ac55d92aefae38f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585daaec60a245308ac55d92aefae38f already in progress: creating tablet
W20260924 19:58:45.503338 20270 catalog_manager.cc:5043] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): delete failed for tablet eb59c659ddf1497182e2fe51c5c68c51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb59c659ddf1497182e2fe51c5c68c51 already in progress: creating tablet
W20260924 19:58:45.506709 20270 catalog_manager.cc:5043] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): delete failed for tablet 585daaec60a245308ac55d92aefae38f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585daaec60a245308ac55d92aefae38f already in progress: creating tablet
I20260924 19:58:45.510417 20278 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_d74dff5eb61a4b6fb10a13c1ec755df9" } modify_external_catalogs: true
I20260924 19:58:45.511349 20278 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_d74dff5eb61a4b6fb10a13c1ec755df9" } modify_external_catalogs: true
I20260924 19:58:45.519301 20977 tablet_bootstrap.cc:654] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:45.522147 20977 log.cc:826] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622: Log is configured to *not* fsync() on all Append() calls
W20260924 19:58:45.528314 20284 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_14cbcbeb7f98417197bb13b1f8b5af49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:45.538651 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 585daaec60a245308ac55d92aefae38f with delete_type TABLET_DATA_DELETED (replaced by tablet 1592544a94794c81ab2f05d91ac67200) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:45.541512 20976 tablet_bootstrap.cc:492] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:45.542111 20976 ts_tablet_manager.cc:1403] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.141s user 0.009s sys 0.005s
I20260924 19:58:45.543478 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet eb59c659ddf1497182e2fe51c5c68c51 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c40348984314595b6e6e49c4f03f35e) from {username='slave'} at 127.0.0.1:60830
W20260924 19:58:45.552835 20280 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_08eddf6c85b94591a21958b5f18ada13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:45.562563 20978 raft_consensus.cc:359] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:45.566036 20978 raft_consensus.cc:385] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:45.566479 20978 raft_consensus.cc:740] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
W20260924 19:58:45.569227 20271 catalog_manager.cc:5043] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet 585daaec60a245308ac55d92aefae38f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 585daaec60a245308ac55d92aefae38f already in progress: creating tablet
I20260924 19:58:45.569346 20555 tablet_service.cc:1511] Processing CreateTablet for tablet 0c40348984314595b6e6e49c4f03f35e (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.569624 20554 tablet_service.cc:1511] Processing CreateTablet for tablet 1592544a94794c81ab2f05d91ac67200 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:58:45.569986 20271 catalog_manager.cc:5043] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet eb59c659ddf1497182e2fe51c5c68c51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb59c659ddf1497182e2fe51c5c68c51 already in progress: creating tablet
I20260924 19:58:45.569803 20282 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_f25b80aa63f9428d849b3723d9f25a39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:45.570468 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c40348984314595b6e6e49c4f03f35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.571050 20554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1592544a94794c81ab2f05d91ac67200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.572206 20847 tablet_service.cc:1511] Processing CreateTablet for tablet 0c40348984314595b6e6e49c4f03f35e (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.573310 20847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c40348984314595b6e6e49c4f03f35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.567734 20702 tablet_service.cc:1511] Processing CreateTablet for tablet 0c40348984314595b6e6e49c4f03f35e (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.580811 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c40348984314595b6e6e49c4f03f35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.586342 20701 tablet_service.cc:1511] Processing CreateTablet for tablet 1592544a94794c81ab2f05d91ac67200 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.587493 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1592544a94794c81ab2f05d91ac67200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.589659 20976 raft_consensus.cc:359] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:45.590567 20976 raft_consensus.cc:385] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:45.590870 20976 raft_consensus.cc:740] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:45.591889 20976 consensus_queue.cc:260] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:45.593230 20848 tablet_service.cc:1511] Processing CreateTablet for tablet 1592544a94794c81ab2f05d91ac67200 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.594287 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1592544a94794c81ab2f05d91ac67200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.592113 20978 consensus_queue.cc:260] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:45.611990 20976 ts_tablet_manager.cc:1434] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.070s user 0.032s sys 0.020s
I20260924 19:58:45.613358 20976 tablet_bootstrap.cc:492] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:45.626725 20978 ts_tablet_manager.cc:1434] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.145s user 0.029s sys 0.005s
I20260924 19:58:45.626935 20976 tablet_bootstrap.cc:654] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:45.627923 20978 tablet_bootstrap.cc:492] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:45.636507 20978 tablet_bootstrap.cc:654] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:45.637028 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 585daaec60a245308ac55d92aefae38f with delete_type TABLET_DATA_DELETED (replaced by tablet 1592544a94794c81ab2f05d91ac67200) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:45.652164 20981 tablet_replica.cc:333] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:45.652995 20981 raft_consensus.cc:2243] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:45.653153 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 585daaec60a245308ac55d92aefae38f with delete_type TABLET_DATA_DELETED (replaced by tablet 1592544a94794c81ab2f05d91ac67200) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:45.653571 20981 raft_consensus.cc:2272] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:45.653805 20977 tablet_bootstrap.cc:492] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:45.654397 20982 tablet_replica.cc:333] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:45.655506 20982 raft_consensus.cc:2243] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:45.656234 20982 raft_consensus.cc:2272] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:45.656538 20981 ts_tablet_manager.cc:1916] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:45.659971 20982 ts_tablet_manager.cc:1916] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:45.662662 20977 ts_tablet_manager.cc:1403] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.231s user 0.023s sys 0.026s
I20260924 19:58:45.681527 20978 tablet_bootstrap.cc:492] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:45.682094 20978 ts_tablet_manager.cc:1403] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.004s
I20260924 19:58:45.684320 20976 tablet_bootstrap.cc:492] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:45.684805 20976 ts_tablet_manager.cc:1403] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.072s user 0.026s sys 0.031s
I20260924 19:58:45.686924 20978 raft_consensus.cc:359] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.691974 20982 ts_tablet_manager.cc:1929] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:45.692432 20982 log.cc:1199] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/585daaec60a245308ac55d92aefae38f
I20260924 19:58:45.693341 20982 ts_tablet_manager.cc:1950] T 585daaec60a245308ac55d92aefae38f P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:45.699321 20978 raft_consensus.cc:385] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:45.699754 20978 raft_consensus.cc:740] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:45.700700 20978 consensus_queue.cc:260] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.703125 20978 ts_tablet_manager.cc:1434] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260924 19:58:45.704412 20978 tablet_bootstrap.cc:492] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:45.710963 20279 master_service.cc:960] ReplaceTablet: received request to replace tablet 585daaec60a245308ac55d92aefae38f from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:45.713305 20277 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_6ac8d98b233146c495713e0cda0ac6b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:45.714043 20978 tablet_bootstrap.cc:654] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:45.717912 20279 master_service.cc:960] ReplaceTablet: received request to replace tablet 585daaec60a245308ac55d92aefae38f from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:45.721134 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet eb59c659ddf1497182e2fe51c5c68c51 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c40348984314595b6e6e49c4f03f35e) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:45.687847 20976 raft_consensus.cc:359] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.727128 20976 raft_consensus.cc:385] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:45.727285 20554 tablet_service.cc:1511] Processing CreateTablet for tablet 635f4961769f49dbb908af7904d29042 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.727558 20976 raft_consensus.cc:740] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:45.728446 20554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 635f4961769f49dbb908af7904d29042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.728359 20976 consensus_queue.cc:260] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.730320 20976 ts_tablet_manager.cc:1434] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.045s user 0.002s sys 0.002s
I20260924 19:58:45.731240 20976 tablet_bootstrap.cc:492] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:45.739665 20702 tablet_service.cc:1511] Processing CreateTablet for tablet 635f4961769f49dbb908af7904d29042 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.740813 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 635f4961769f49dbb908af7904d29042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.741333 20976 tablet_bootstrap.cc:654] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:45.746646 20846 tablet_service.cc:1511] Processing CreateTablet for tablet 635f4961769f49dbb908af7904d29042 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:45.748034 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 635f4961769f49dbb908af7904d29042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:45.759018 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 585daaec60a245308ac55d92aefae38f (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:45.763439 20845 tablet_service.cc:1558] Processing DeleteTablet for tablet eb59c659ddf1497182e2fe51c5c68c51 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c40348984314595b6e6e49c4f03f35e) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:45.771482 20982 tablet_replica.cc:333] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:45.779743 20982 raft_consensus.cc:2243] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:45.780375 20982 raft_consensus.cc:2272] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:45.782749 20982 ts_tablet_manager.cc:1916] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:45.771559 20976 tablet_bootstrap.cc:492] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:45.791754 20976 ts_tablet_manager.cc:1403] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.061s user 0.019s sys 0.014s
I20260924 19:58:45.794461 20976 raft_consensus.cc:359] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.795199 20976 raft_consensus.cc:385] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:45.795487 20976 raft_consensus.cc:740] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:45.802559 20976 consensus_queue.cc:260] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.805047 20981 ts_tablet_manager.cc:1929] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:45.805493 20981 log.cc:1199] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/585daaec60a245308ac55d92aefae38f
I20260924 19:58:45.807993 20978 tablet_bootstrap.cc:492] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:45.808473 20978 ts_tablet_manager.cc:1403] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.104s user 0.013s sys 0.015s
I20260924 19:58:45.813959 20978 raft_consensus.cc:359] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.814695 20978 raft_consensus.cc:385] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:45.814978 20978 raft_consensus.cc:740] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:45.815708 20978 consensus_queue.cc:260] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.817234 20981 ts_tablet_manager.cc:1950] T 585daaec60a245308ac55d92aefae38f P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:45.825659 20981 tablet_replica.cc:333] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:45.825855 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 585daaec60a245308ac55d92aefae38f (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:45.826323 20981 raft_consensus.cc:2243] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:45.826756 20981 raft_consensus.cc:2272] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:45.828414 20978 ts_tablet_manager.cc:1434] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260924 19:58:45.828872 20976 ts_tablet_manager.cc:1434] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20260924 19:58:45.833685 20978 tablet_bootstrap.cc:492] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:45.841611 20976 tablet_bootstrap.cc:492] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:45.846992 20978 tablet_bootstrap.cc:654] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:45.850534 20982 ts_tablet_manager.cc:1929] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:45.853004 20977 raft_consensus.cc:359] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:45.853987 20977 raft_consensus.cc:385] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:45.854315 20977 raft_consensus.cc:740] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:45.854163 20982 log.cc:1199] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/eb59c659ddf1497182e2fe51c5c68c51
I20260924 19:58:45.855353 20977 consensus_queue.cc:260] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:45.856694 20982 ts_tablet_manager.cc:1950] T eb59c659ddf1497182e2fe51c5c68c51 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:45.857481 20981 ts_tablet_manager.cc:1916] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:45.865756 20976 tablet_bootstrap.cc:654] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
W20260924 19:58:45.867594 20776 tablet.cc:2433] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:58:45.868232 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet eb59c659ddf1497182e2fe51c5c68c51 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:45.877508 20977 ts_tablet_manager.cc:1434] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.215s user 0.033s sys 0.008s
I20260924 19:58:45.886420 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet 585daaec60a245308ac55d92aefae38f with delete_type TABLET_DATA_DELETED (replaced by tablet 1592544a94794c81ab2f05d91ac67200) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:45.887454 20980 tablet_replica.cc:333] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:45.888315 20980 raft_consensus.cc:2243] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:45.888885 20978 tablet_bootstrap.cc:492] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:45.888883 20980 raft_consensus.cc:2272] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:45.889314 20978 ts_tablet_manager.cc:1403] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.005s
I20260924 19:58:45.891553 20977 tablet_bootstrap.cc:492] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:45.892985 20978 raft_consensus.cc:359] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.893683 20978 raft_consensus.cc:385] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:45.893986 20978 raft_consensus.cc:740] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:45.894711 20978 consensus_queue.cc:260] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.896826 20978 ts_tablet_manager.cc:1434] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 19:58:45.901659 20977 tablet_bootstrap.cc:654] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:45.906528 20980 ts_tablet_manager.cc:1916] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:45.907059 20978 tablet_bootstrap.cc:492] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:45.913511 20976 tablet_bootstrap.cc:492] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:45.913959 20976 ts_tablet_manager.cc:1403] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.001s
I20260924 19:58:45.917968 20976 raft_consensus.cc:359] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.918614 20976 raft_consensus.cc:385] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:45.918891 20976 raft_consensus.cc:740] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:45.929353 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c40348984314595b6e6e49c4f03f35e from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:45.930840 20978 tablet_bootstrap.cc:654] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:45.933454 20976 consensus_queue.cc:260] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.935858 20981 ts_tablet_manager.cc:1929] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:45.936234 20981 log.cc:1199] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/eb59c659ddf1497182e2fe51c5c68c51
I20260924 19:58:45.936890 20981 ts_tablet_manager.cc:1950] T eb59c659ddf1497182e2fe51c5c68c51 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:45.931005 20275 master_service.cc:960] ReplaceTablet: received request to replace tablet 635f4961769f49dbb908af7904d29042 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:45.946491 20977 tablet_bootstrap.cc:492] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:45.947055 20977 ts_tablet_manager.cc:1403] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.012s
I20260924 19:58:45.949110 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet eb59c659ddf1497182e2fe51c5c68c51 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:45.950052 20977 raft_consensus.cc:359] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.950809 20977 raft_consensus.cc:385] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:45.951140 20977 raft_consensus.cc:740] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:45.952160 20977 consensus_queue.cc:260] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:45.954696 20276 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 0c40348984314595b6e6e49c4f03f35e
I20260924 19:58:45.945832 20980 ts_tablet_manager.cc:1929] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:45.955381 20980 log.cc:1199] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/585daaec60a245308ac55d92aefae38f
I20260924 19:58:45.956264 20276 catalog_manager.cc:6934] ReplaceTablet: tablet 0c40348984314595b6e6e49c4f03f35e deleted and replaced by tablet 3647da617a934fb69e70c056a7a88f47
I20260924 19:58:45.956359 20980 ts_tablet_manager.cc:1950] T 585daaec60a245308ac55d92aefae38f P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:45.958078 20977 ts_tablet_manager.cc:1434] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260924 19:58:45.959059 20977 tablet_bootstrap.cc:492] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:45.959563 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c40348984314595b6e6e49c4f03f35e with delete_type TABLET_DATA_DELETED (replaced by tablet 3647da617a934fb69e70c056a7a88f47) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:45.960891 20981 tablet_replica.cc:333] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:45.960949 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c40348984314595b6e6e49c4f03f35e with delete_type TABLET_DATA_DELETED (replaced by tablet 3647da617a934fb69e70c056a7a88f47) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:45.961621 20981 raft_consensus.cc:2243] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:45.961680 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c40348984314595b6e6e49c4f03f35e with delete_type TABLET_DATA_DELETED (replaced by tablet 3647da617a934fb69e70c056a7a88f47) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:45.962116 20981 raft_consensus.cc:2272] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:45.962926 20982 tablet_replica.cc:333] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:45.963639 20982 raft_consensus.cc:2243] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:45.969460 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 585daaec60a245308ac55d92aefae38f (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
W20260924 19:58:45.970945 20271 catalog_manager.cc:5043] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet 0c40348984314595b6e6e49c4f03f35e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c40348984314595b6e6e49c4f03f35e already in progress: creating tablet
I20260924 19:58:45.972174 20981 ts_tablet_manager.cc:1916] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:45.972859 20977 tablet_bootstrap.cc:654] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:45.982698 20275 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:45.983419 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet eb59c659ddf1497182e2fe51c5c68c51 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c40348984314595b6e6e49c4f03f35e) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:45.984194 20980 tablet_replica.cc:333] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:45.984874 20980 raft_consensus.cc:2243] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:45.985340 20980 raft_consensus.cc:2272] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:45.986074 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 635f4961769f49dbb908af7904d29042 with delete_type TABLET_DATA_DELETED (replaced by tablet a7a74e99149347358597fcf67fce2791) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:45.986236 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 635f4961769f49dbb908af7904d29042 with delete_type TABLET_DATA_DELETED (replaced by tablet a7a74e99149347358597fcf67fce2791) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:45.987049 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet 635f4961769f49dbb908af7904d29042 with delete_type TABLET_DATA_DELETED (replaced by tablet a7a74e99149347358597fcf67fce2791) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:45.988945 20980 ts_tablet_manager.cc:1916] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:45.991297 20275 catalog_manager.cc:6934] ReplaceTablet: tablet 635f4961769f49dbb908af7904d29042 deleted and replaced by tablet a7a74e99149347358597fcf67fce2791
I20260924 19:58:45.999225 20982 raft_consensus.cc:2272] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:46.002981 20982 ts_tablet_manager.cc:1916] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:58:46.009282 20959 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:58:46.010120 20959 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:58:46.010902 20959 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260924 19:58:46.018066 20981 ts_tablet_manager.cc:1929] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:46.018473 20981 log.cc:1199] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/0c40348984314595b6e6e49c4f03f35e
I20260924 19:58:46.019181 20981 ts_tablet_manager.cc:1950] T 0c40348984314595b6e6e49c4f03f35e P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:46.020673 20977 tablet_bootstrap.cc:492] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:46.021123 20977 ts_tablet_manager.cc:1403] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.010s
I20260924 19:58:46.021854 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 0c40348984314595b6e6e49c4f03f35e (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:46.023671 20977 raft_consensus.cc:359] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.024364 20977 raft_consensus.cc:385] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.024663 20977 raft_consensus.cc:740] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.025542 20977 consensus_queue.cc:260] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.027335 20554 tablet_service.cc:1511] Processing CreateTablet for tablet a7a74e99149347358597fcf67fce2791 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.028748 20554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a74e99149347358597fcf67fce2791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.030176 20976 ts_tablet_manager.cc:1434] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.116s user 0.008s sys 0.000s
I20260924 19:58:46.030076 20702 tablet_service.cc:1511] Processing CreateTablet for tablet a7a74e99149347358597fcf67fce2791 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.031217 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a74e99149347358597fcf67fce2791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.038219 20701 tablet_service.cc:1511] Processing CreateTablet for tablet 3647da617a934fb69e70c056a7a88f47 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.035828 20846 tablet_service.cc:1511] Processing CreateTablet for tablet a7a74e99149347358597fcf67fce2791 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.039928 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a74e99149347358597fcf67fce2791. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.040062 20976 tablet_bootstrap.cc:492] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:46.042100 20977 ts_tablet_manager.cc:1434] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20260924 19:58:46.046747 20555 tablet_service.cc:1511] Processing CreateTablet for tablet 3647da617a934fb69e70c056a7a88f47 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.047919 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3647da617a934fb69e70c056a7a88f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.049168 20977 tablet_bootstrap.cc:492] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:46.056877 20980 ts_tablet_manager.cc:1929] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:46.057281 20980 log.cc:1199] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/eb59c659ddf1497182e2fe51c5c68c51
I20260924 19:58:46.057973 20980 ts_tablet_manager.cc:1950] T eb59c659ddf1497182e2fe51c5c68c51 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:46.061237 20987 raft_consensus.cc:493] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:46.061821 20987 raft_consensus.cc:515] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.063783 20848 tablet_service.cc:1511] Processing CreateTablet for tablet 3647da617a934fb69e70c056a7a88f47 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.064913 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3647da617a934fb69e70c056a7a88f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.061369 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3647da617a934fb69e70c056a7a88f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.073405 20982 ts_tablet_manager.cc:1929] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:46.073871 20982 log.cc:1199] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/0c40348984314595b6e6e49c4f03f35e
I20260924 19:58:46.074270 20986 raft_consensus.cc:493] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:46.074783 20987 leader_election.cc:290] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:46.074827 20986 raft_consensus.cc:515] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:58:46.076172 20276 catalog_manager.cc:5671] Ignoring report from unknown tablet 0c40348984314595b6e6e49c4f03f35e
I20260924 19:58:46.079492 20986 leader_election.cc:290] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:46.088090 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet eb59c659ddf1497182e2fe51c5c68c51 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:46.088268 20982 ts_tablet_manager.cc:1950] T 0c40348984314595b6e6e49c4f03f35e P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
W20260924 19:58:46.092312 20271 catalog_manager.cc:5043] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet 635f4961769f49dbb908af7904d29042 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 635f4961769f49dbb908af7904d29042 already in progress: creating tablet
W20260924 19:58:46.094779 20275 catalog_manager.cc:5671] Ignoring report from unknown tablet 0c40348984314595b6e6e49c4f03f35e
I20260924 19:58:46.100449 20978 tablet_bootstrap.cc:492] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:46.100905 20978 ts_tablet_manager.cc:1403] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.194s user 0.010s sys 0.000s
I20260924 19:58:46.106801 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 0c40348984314595b6e6e49c4f03f35e (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:46.113497 20977 tablet_bootstrap.cc:654] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.115271 20976 tablet_bootstrap.cc:654] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
W20260924 19:58:46.115469 20270 catalog_manager.cc:5043] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): delete failed for tablet 635f4961769f49dbb908af7904d29042 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 635f4961769f49dbb908af7904d29042 already in progress: creating tablet
I20260924 19:58:46.122618 20978 raft_consensus.cc:359] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.123368 20978 raft_consensus.cc:385] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.123667 20978 raft_consensus.cc:740] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.124377 20978 consensus_queue.cc:260] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:58:46.133170 20271 catalog_manager.cc:5043] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet 635f4961769f49dbb908af7904d29042 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 635f4961769f49dbb908af7904d29042 already in progress: creating tablet
W20260924 19:58:46.145426 20276 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:46.152143 20978 ts_tablet_manager.cc:1434] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.051s user 0.007s sys 0.001s
I20260924 19:58:46.153235 20978 tablet_bootstrap.cc:492] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:46.168840 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 635f4961769f49dbb908af7904d29042 from {username='slave'} at 127.0.0.1:40798
W20260924 19:58:46.185285 20276 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:46.187810 20997 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:46.188403 20997 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.190376 20976 tablet_bootstrap.cc:492] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:46.190874 20976 ts_tablet_manager.cc:1403] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.151s user 0.009s sys 0.008s
I20260924 19:58:46.191880 20997 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:46.201344 20977 tablet_bootstrap.cc:492] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:46.201802 20977 ts_tablet_manager.cc:1403] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.153s user 0.006s sys 0.014s
I20260924 19:58:46.202406 20976 raft_consensus.cc:359] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.204668 20976 raft_consensus.cc:385] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.204512 20977 raft_consensus.cc:359] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.205200 20977 raft_consensus.cc:385] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.205482 20977 raft_consensus.cc:740] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.205372 20976 raft_consensus.cc:740] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.210206 20976 consensus_queue.cc:260] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:58:46.214576 20918 tablet.cc:2433] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:58:46.216480 20978 tablet_bootstrap.cc:654] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
W20260924 19:58:46.219312 20276 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:46.219627 20977 consensus_queue.cc:260] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.222098 20977 ts_tablet_manager.cc:1434] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260924 19:58:46.223089 20977 tablet_bootstrap.cc:492] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:46.235229 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c40348984314595b6e6e49c4f03f35e from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:46.235769 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1592544a94794c81ab2f05d91ac67200" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:46.236537 20575 raft_consensus.cc:2468] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:46.238062 20658 leader_election.cc:304] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:46.239189 20987 raft_consensus.cc:2804] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:46.239581 20987 raft_consensus.cc:493] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:46.239929 20987 raft_consensus.cc:3060] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:46.244050 20976 ts_tablet_manager.cc:1434] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.053s user 0.003s sys 0.003s
I20260924 19:58:46.247040 20987 raft_consensus.cc:515] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.249876 20987 leader_election.cc:290] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:46.251044 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1592544a94794c81ab2f05d91ac67200" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:46.251223 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1592544a94794c81ab2f05d91ac67200" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:46.251787 20575 raft_consensus.cc:3060] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:58:46.252000 20276 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:46.252055 20722 raft_consensus.cc:2393] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0055218dbcbd48b58018e9774e59e14c in current term 1: Already voted for candidate af20c71727a14e398a217d4623d1399a in this term.
I20260924 19:58:46.257421 20868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1592544a94794c81ab2f05d91ac67200" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:46.258219 20868 raft_consensus.cc:2468] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
W20260924 19:58:46.259888 20623 tablet.cc:2433] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:58:46.259742 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1592544a94794c81ab2f05d91ac67200" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:46.260372 20867 raft_consensus.cc:3060] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:46.268055 20867 raft_consensus.cc:2468] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:46.269367 20657 leader_election.cc:304] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:46.270198 20987 raft_consensus.cc:2804] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:46.269745 20976 tablet_bootstrap.cc:492] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:46.263325 20575 raft_consensus.cc:2468] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:46.282011 20987 raft_consensus.cc:697] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Becoming Leader. State: Replica: af20c71727a14e398a217d4623d1399a, State: Running, Role: LEADER
I20260924 19:58:46.282012 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1592544a94794c81ab2f05d91ac67200" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:46.282835 20575 raft_consensus.cc:2393] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0055218dbcbd48b58018e9774e59e14c in current term 1: Already voted for candidate af20c71727a14e398a217d4623d1399a in this term.
I20260924 19:58:46.286473 20804 leader_election.cc:304] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:58:46.282953 20987 consensus_queue.cc:237] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.287567 20976 tablet_bootstrap.cc:654] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.287745 20986 raft_consensus.cc:2749] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:46.292960 20978 tablet_bootstrap.cc:492] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:46.293401 20978 ts_tablet_manager.cc:1403] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.140s user 0.007s sys 0.010s
W20260924 19:58:46.293615 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 19:58:46.299001 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:58:46.299481 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:58:46.300297 20997 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:46.301842 20978 raft_consensus.cc:359] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:46.302524 20978 raft_consensus.cc:385] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.302803 20978 raft_consensus.cc:740] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.303496 20978 consensus_queue.cc:260] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:46.315716 20978 ts_tablet_manager.cc:1434] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.022s user 0.005s sys 0.002s
I20260924 19:58:46.319895 20978 tablet_bootstrap.cc:492] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:46.320495 20977 tablet_bootstrap.cc:654] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.330470 20976 tablet_bootstrap.cc:492] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:46.330940 20976 ts_tablet_manager.cc:1403] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.006s
I20260924 19:58:46.337711 20276 catalog_manager.cc:5838] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a reported cstate change: term changed from 0 to 1, leader changed from <none> to af20c71727a14e398a217d4623d1399a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "af20c71727a14e398a217d4623d1399a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20260924 19:58:46.342980 20976 raft_consensus.cc:359] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:46.343675 20976 raft_consensus.cc:385] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.343969 20976 raft_consensus.cc:740] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.353896 20977 tablet_bootstrap.cc:492] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:46.354343 20977 ts_tablet_manager.cc:1403] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.131s user 0.011s sys 0.000s
I20260924 19:58:46.355487 20976 consensus_queue.cc:260] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:46.357082 20977 raft_consensus.cc:359] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.357751 20977 raft_consensus.cc:385] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.358037 20977 raft_consensus.cc:740] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.364818 20976 ts_tablet_manager.cc:1434] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260924 19:58:46.365749 20976 tablet_bootstrap.cc:492] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:46.372049 20977 consensus_queue.cc:260] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.374799 20978 tablet_bootstrap.cc:654] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.379048 20275 master_service.cc:960] ReplaceTablet: received request to replace tablet 1592544a94794c81ab2f05d91ac67200 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:46.380772 20977 ts_tablet_manager.cc:1434] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260924 19:58:46.384442 20976 tablet_bootstrap.cc:654] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.388065 20987 raft_consensus.cc:493] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:46.388531 20987 raft_consensus.cc:515] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
W20260924 19:58:46.389479 20279 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:46.390538 20987 leader_election.cc:290] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:46.392539 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a74e99149347358597fcf67fce2791" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:46.392899 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a74e99149347358597fcf67fce2791" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:46.393492 20867 raft_consensus.cc:2468] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
W20260924 19:58:46.394064 20658 leader_election.cc:343] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:46.394728 20657 leader_election.cc:304] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters: 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:58:46.381700 20977 tablet_bootstrap.cc:492] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
W20260924 19:58:46.401407 20959 flags.cc:432] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260924 19:58:46.401749 20959 flags.cc:432] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260924 19:58:46.402092 20959 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
I20260924 19:58:46.402312 20987 raft_consensus.cc:2804] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260924 19:58:46.402464 20959 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
I20260924 19:58:46.402667 20987 raft_consensus.cc:493] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 19:58:46.402810 20959 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260924 19:58:46.403010 20987 raft_consensus.cc:3060] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:58:46.403137 20959 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:58:46.420889 20275 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 1592544a94794c81ab2f05d91ac67200
I20260924 19:58:46.422505 20275 catalog_manager.cc:6934] ReplaceTablet: tablet 1592544a94794c81ab2f05d91ac67200 deleted and replaced by tablet d26c86130a67479d9f8ef3a829f8f6a8
I20260924 19:58:46.423286 20701 tablet_service.cc:1558] Processing DeleteTablet for tablet 1592544a94794c81ab2f05d91ac67200 with delete_type TABLET_DATA_DELETED (replaced by tablet d26c86130a67479d9f8ef3a829f8f6a8) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:46.424072 20981 tablet_replica.cc:333] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:46.424784 20981 raft_consensus.cc:2243] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:46.425132 20977 tablet_bootstrap.cc:654] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.425566 20981 pending_rounds.cc:70] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: Trying to abort 1 pending ops.
I20260924 19:58:46.425837 20981 pending_rounds.cc:77] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986578105716736 op_type: NO_OP noop_request { }
I20260924 19:58:46.425738 20987 raft_consensus.cc:515] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:46.426363 20981 raft_consensus.cc:2889] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:58:46.426725 20981 raft_consensus.cc:2272] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:46.427456 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet 1592544a94794c81ab2f05d91ac67200 with delete_type TABLET_DATA_DELETED (replaced by tablet d26c86130a67479d9f8ef3a829f8f6a8) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:46.427578 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet 1592544a94794c81ab2f05d91ac67200 with delete_type TABLET_DATA_DELETED (replaced by tablet d26c86130a67479d9f8ef3a829f8f6a8) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:46.428174 20980 tablet_replica.cc:333] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
W20260924 19:58:46.428416 20276 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:46.428920 20980 raft_consensus.cc:2243] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:46.429400 20980 raft_consensus.cc:2272] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:46.429916 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a74e99149347358597fcf67fce2791" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:46.430496 20867 raft_consensus.cc:3060] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:46.432509 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a74e99149347358597fcf67fce2791" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:46.430169 20982 tablet_replica.cc:333] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:46.433923 20982 raft_consensus.cc:2243] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:46.434393 20982 raft_consensus.cc:2272] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:58:46.435791 20658 leader_election.cc:343] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:46.436787 20867 raft_consensus.cc:2468] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:46.437760 20980 ts_tablet_manager.cc:1916] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:46.437908 20657 leader_election.cc:304] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters: 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:58:46.438812 21019 raft_consensus.cc:2804] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:46.444005 20275 master_service.cc:960] ReplaceTablet: received request to replace tablet a7a74e99149347358597fcf67fce2791 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:46.445082 20981 ts_tablet_manager.cc:1916] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:46.446815 20987 leader_election.cc:290] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:46.448522 20982 ts_tablet_manager.cc:1916] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:46.464444 20978 tablet_bootstrap.cc:492] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:46.464879 20978 ts_tablet_manager.cc:1403] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.145s user 0.005s sys 0.007s
I20260924 19:58:46.471247 20848 tablet_service.cc:1511] Processing CreateTablet for tablet d26c86130a67479d9f8ef3a829f8f6a8 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.472121 20701 tablet_service.cc:1511] Processing CreateTablet for tablet d26c86130a67479d9f8ef3a829f8f6a8 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.472568 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26c86130a67479d9f8ef3a829f8f6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.473203 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26c86130a67479d9f8ef3a829f8f6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.475145 20281 catalog_manager.cc:3541] renamed table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29] in HMS: new name default.table_92546dd5153b4d58af6fc19d4dee39f0
I20260924 19:58:46.480145 20555 tablet_service.cc:1511] Processing CreateTablet for tablet d26c86130a67479d9f8ef3a829f8f6a8 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.481303 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26c86130a67479d9f8ef3a829f8f6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.482909 21019 raft_consensus.cc:697] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Becoming Leader. State: Replica: af20c71727a14e398a217d4623d1399a, State: Running, Role: LEADER
I20260924 19:58:46.483772 21019 consensus_queue.cc:237] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:46.487238 20978 raft_consensus.cc:359] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.487875 20978 raft_consensus.cc:385] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.488114 20978 raft_consensus.cc:740] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.495431 20978 consensus_queue.cc:260] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.497368 20978 ts_tablet_manager.cc:1434] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
I20260924 19:58:46.498602 20986 raft_consensus.cc:493] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:46.499277 20986 raft_consensus.cc:515] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.501169 20986 leader_election.cc:290] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:46.503288 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "635f4961769f49dbb908af7904d29042" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:46.503912 20575 raft_consensus.cc:2468] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
I20260924 19:58:46.505877 20804 leader_election.cc:304] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters:
I20260924 19:58:46.506701 20986 raft_consensus.cc:2804] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:46.507051 20986 raft_consensus.cc:493] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:46.507396 20986 raft_consensus.cc:3060] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:46.508502 20977 tablet_bootstrap.cc:492] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:46.508982 20977 ts_tablet_manager.cc:1403] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.127s user 0.010s sys 0.002s
I20260924 19:58:46.511736 20977 raft_consensus.cc:359] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:46.512403 20977 raft_consensus.cc:385] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.512694 20977 raft_consensus.cc:740] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.513393 20977 consensus_queue.cc:260] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:46.514371 20986 raft_consensus.cc:515] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.516786 20976 tablet_bootstrap.cc:492] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:46.517184 20976 ts_tablet_manager.cc:1403] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.152s user 0.013s sys 0.014s
I20260924 19:58:46.519824 20976 raft_consensus.cc:359] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.520486 20976 raft_consensus.cc:385] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.520789 20976 raft_consensus.cc:740] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.521605 20976 consensus_queue.cc:260] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.525172 20982 ts_tablet_manager.cc:1929] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:46.525185 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "635f4961769f49dbb908af7904d29042" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:46.525560 20982 log.cc:1199] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/1592544a94794c81ab2f05d91ac67200
I20260924 19:58:46.525790 20575 raft_consensus.cc:3060] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:46.526264 20982 ts_tablet_manager.cc:1950] T 1592544a94794c81ab2f05d91ac67200 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:46.528268 20275 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet a7a74e99149347358597fcf67fce2791
I20260924 19:58:46.528967 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 1592544a94794c81ab2f05d91ac67200 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:46.530138 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet a7a74e99149347358597fcf67fce2791 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a12482cbc69483d8789f530de3393e1) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:46.531137 20977 ts_tablet_manager.cc:1434] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260924 19:58:46.533301 20575 raft_consensus.cc:2468] T 635f4961769f49dbb908af7904d29042 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 1.
I20260924 19:58:46.534505 20804 leader_election.cc:304] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters:
I20260924 19:58:46.537714 20977 tablet_bootstrap.cc:492] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:46.540838 20976 ts_tablet_manager.cc:1434] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.023s user 0.005s sys 0.006s
I20260924 19:58:46.543798 20976 tablet_bootstrap.cc:492] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:46.545923 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "635f4961769f49dbb908af7904d29042" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:46.546155 20981 ts_tablet_manager.cc:1929] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:46.546518 20981 log.cc:1199] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/1592544a94794c81ab2f05d91ac67200
I20260924 19:58:46.546602 20722 raft_consensus.cc:2468] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
I20260924 19:58:46.547341 20981 ts_tablet_manager.cc:1950] T 1592544a94794c81ab2f05d91ac67200 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:46.550107 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 1592544a94794c81ab2f05d91ac67200 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:46.558255 20997 raft_consensus.cc:493] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:46.558750 20997 raft_consensus.cc:515] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:46.559544 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "635f4961769f49dbb908af7904d29042" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
I20260924 19:58:46.560082 20722 raft_consensus.cc:3060] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:46.560683 20997 leader_election.cc:290] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:46.563532 20721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a74e99149347358597fcf67fce2791" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:46.564635 20977 tablet_bootstrap.cc:654] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.575870 20722 raft_consensus.cc:2468] T 635f4961769f49dbb908af7904d29042 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 1.
I20260924 19:58:46.587790 20275 catalog_manager.cc:6934] ReplaceTablet: tablet a7a74e99149347358597fcf67fce2791 deleted and replaced by tablet 1a12482cbc69483d8789f530de3393e1
I20260924 19:58:46.589037 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet a7a74e99149347358597fcf67fce2791 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a12482cbc69483d8789f530de3393e1) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:46.589898 20981 tablet_replica.cc:333] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:46.590767 20981 raft_consensus.cc:2243] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:46.591629 20981 pending_rounds.cc:70] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: Trying to abort 1 pending ops.
I20260924 19:58:46.591904 20981 pending_rounds.cc:77] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986578888851456 op_type: NO_OP noop_request { }
I20260924 19:58:46.592327 20981 raft_consensus.cc:2889] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:58:46.592737 20981 raft_consensus.cc:2272] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:46.594383 20959 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.2.148:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:33597
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.2.188:35311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.2.188
--webserver_port=36753
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision ae52405967cd2709483ea5c02e613be50ab23f43
build type FASTDEBUG
built by None at 24 Sep 2026 19:43:20 UTC on e7f111948823
build id 13982
TSAN enabled
I20260924 19:58:46.560968 20976 tablet_bootstrap.cc:654] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.556337 20986 leader_election.cc:290] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:46.597958 21029 raft_consensus.cc:2804] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:46.564621 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7a74e99149347358597fcf67fce2791" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:46.600519 20867 raft_consensus.cc:2393] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 30a4f50fc28c4616ba622ba51b0b2622 in current term 1: Already voted for candidate af20c71727a14e398a217d4623d1399a in this term.
I20260924 19:58:46.581426 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet a7a74e99149347358597fcf67fce2791 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a12482cbc69483d8789f530de3393e1) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:46.602432 20982 tablet_replica.cc:333] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:46.603106 20959 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:58:46.603231 20982 raft_consensus.cc:2243] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:46.603696 20982 raft_consensus.cc:2272] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:46.605685 20982 ts_tablet_manager.cc:1916] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:46.605794 20959 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260924 19:58:46.606805 20508 leader_election.cc:304] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:58:46.607674 20997 raft_consensus.cc:3060] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:46.614660 20997 raft_consensus.cc:2749] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:46.615809 21019 raft_consensus.cc:493] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:46.616248 21019 raft_consensus.cc:515] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.619223 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3647da617a934fb69e70c056a7a88f47" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
W20260924 19:58:46.620543 20658 leader_election.cc:343] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:46.621908 20980 ts_tablet_manager.cc:1929] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:46.622314 20980 log.cc:1199] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/1592544a94794c81ab2f05d91ac67200
I20260924 19:58:46.622972 20980 ts_tablet_manager.cc:1950] T 1592544a94794c81ab2f05d91ac67200 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:46.625567 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 1592544a94794c81ab2f05d91ac67200 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:46.625576 21029 raft_consensus.cc:697] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: Becoming Leader. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Running, Role: LEADER
I20260924 19:58:46.626550 21029 consensus_queue.cc:237] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.631556 20981 ts_tablet_manager.cc:1916] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:46.633864 20977 tablet_bootstrap.cc:492] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:46.634325 20977 ts_tablet_manager.cc:1403] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.102s user 0.015s sys 0.011s
I20260924 19:58:46.634636 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 3647da617a934fb69e70c056a7a88f47 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:46.636888 20977 raft_consensus.cc:359] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.640651 20976 tablet_bootstrap.cc:492] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:46.641003 20977 raft_consensus.cc:385] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.641083 20976 ts_tablet_manager.cc:1403] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.097s user 0.008s sys 0.020s
I20260924 19:58:46.641371 20977 raft_consensus.cc:740] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.643590 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3647da617a934fb69e70c056a7a88f47" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:46.643509 20976 raft_consensus.cc:359] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.644186 20976 raft_consensus.cc:385] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.644289 21019 leader_election.cc:290] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:46.644459 20976 raft_consensus.cc:740] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.644193 20867 raft_consensus.cc:2468] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:46.643440 20977 consensus_queue.cc:260] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.646373 20657 leader_election.cc:304] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters: 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:58:46.646032 20976 consensus_queue.cc:260] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.647722 20982 ts_tablet_manager.cc:1929] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:46.648809 20977 ts_tablet_manager.cc:1434] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.014s user 0.001s sys 0.005s
I20260924 19:58:46.648465 20976 ts_tablet_manager.cc:1434] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 19:58:46.655547 20982 log.cc:1199] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/a7a74e99149347358597fcf67fce2791
I20260924 19:58:46.656864 20982 ts_tablet_manager.cc:1950] T a7a74e99149347358597fcf67fce2791 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:46.660050 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet a7a74e99149347358597fcf67fce2791 (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:46.666911 21019 raft_consensus.cc:2804] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:46.668043 21019 raft_consensus.cc:493] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:46.668412 21019 raft_consensus.cc:3060] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:46.687811 20980 tablet_replica.cc:333] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:46.688616 20980 raft_consensus.cc:2243] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:46.689131 20980 raft_consensus.cc:2272] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:46.713100 20276 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 3647da617a934fb69e70c056a7a88f47
I20260924 19:58:46.715668 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 3647da617a934fb69e70c056a7a88f47 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c7d4336285d4ed2b8252609894f9a49) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:46.716892 20276 catalog_manager.cc:6934] ReplaceTablet: tablet 3647da617a934fb69e70c056a7a88f47 deleted and replaced by tablet 0c7d4336285d4ed2b8252609894f9a49
I20260924 19:58:46.718449 20980 ts_tablet_manager.cc:1916] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:46.721864 20283 catalog_manager.cc:3541] renamed table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0] in HMS: new name default.table_1ed870b0a11e446c86e6a4c6a15e81b6
I20260924 19:58:46.724531 20981 ts_tablet_manager.cc:1929] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:46.724949 20981 log.cc:1199] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/a7a74e99149347358597fcf67fce2791
I20260924 19:58:46.727079 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet 3647da617a934fb69e70c056a7a88f47 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c7d4336285d4ed2b8252609894f9a49) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:46.727854 20982 tablet_replica.cc:333] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:46.728229 21019 raft_consensus.cc:515] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.728623 20982 raft_consensus.cc:2243] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
W20260924 19:58:46.728724 20275 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:46.729084 20982 raft_consensus.cc:2272] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:46.731305 20982 ts_tablet_manager.cc:1916] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:46.735379 20702 tablet_service.cc:1511] Processing CreateTablet for tablet 1a12482cbc69483d8789f530de3393e1 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.736897 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a12482cbc69483d8789f530de3393e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.745929 20982 ts_tablet_manager.cc:1929] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:46.746410 20982 log.cc:1199] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/3647da617a934fb69e70c056a7a88f47
I20260924 19:58:46.746899 21019 leader_election.cc:290] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:46.747220 20982 ts_tablet_manager.cc:1950] T 3647da617a934fb69e70c056a7a88f47 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:46.725724 20981 ts_tablet_manager.cc:1950] T a7a74e99149347358597fcf67fce2791 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:46.749866 20981 tablet_replica.cc:333] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
W20260924 19:58:46.750185 20657 leader_election.cc:343] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 3647da617a934fb69e70c056a7a88f47
I20260924 19:58:46.750582 20981 raft_consensus.cc:2243] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:46.751097 20981 raft_consensus.cc:2272] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:46.751789 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet a7a74e99149347358597fcf67fce2791 (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:46.753341 20981 ts_tablet_manager.cc:1916] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:46.754707 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 3647da617a934fb69e70c056a7a88f47 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:46.769157 20848 tablet_service.cc:1511] Processing CreateTablet for tablet 1a12482cbc69483d8789f530de3393e1 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.770701 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a12482cbc69483d8789f530de3393e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.782240 20978 tablet_bootstrap.cc:492] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:46.785962 20553 tablet_service.cc:1511] Processing CreateTablet for tablet 1a12482cbc69483d8789f530de3393e1 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.787773 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a12482cbc69483d8789f530de3393e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.790407 20978 tablet_bootstrap.cc:654] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.794713 20981 ts_tablet_manager.cc:1929] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:46.795605 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet 3647da617a934fb69e70c056a7a88f47 with delete_type TABLET_DATA_DELETED (replaced by tablet 0c7d4336285d4ed2b8252609894f9a49) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:46.797506 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 635f4961769f49dbb908af7904d29042 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:46.797163 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3647da617a934fb69e70c056a7a88f47" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:46.798220 20575 raft_consensus.cc:3060] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:46.805411 20575 raft_consensus.cc:2468] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:46.806710 20658 leader_election.cc:304] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c
W20260924 19:58:46.807404 20658 raft_consensus.cc:2686] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260924 19:58:46.795095 20981 log.cc:1199] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/3647da617a934fb69e70c056a7a88f47
I20260924 19:58:46.809559 20981 ts_tablet_manager.cc:1950] T 3647da617a934fb69e70c056a7a88f47 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:46.809701 20977 tablet_bootstrap.cc:492] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:46.813287 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 3647da617a934fb69e70c056a7a88f47 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:46.816535 20977 tablet_bootstrap.cc:654] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.840041 20980 ts_tablet_manager.cc:1929] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:46.840505 20980 log.cc:1199] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/a7a74e99149347358597fcf67fce2791
I20260924 19:58:46.841387 20980 ts_tablet_manager.cc:1950] T a7a74e99149347358597fcf67fce2791 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:46.844527 20980 tablet_replica.cc:333] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:46.844874 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet a7a74e99149347358597fcf67fce2791 (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:46.845359 20978 tablet_bootstrap.cc:492] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:46.845338 20980 raft_consensus.cc:2243] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:46.845763 20978 ts_tablet_manager.cc:1403] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.003s
I20260924 19:58:46.845845 20980 raft_consensus.cc:2272] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:46.847117 20977 tablet_bootstrap.cc:492] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:46.847535 20977 ts_tablet_manager.cc:1403] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.006s
I20260924 19:58:46.848024 20978 raft_consensus.cc:359] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.848687 20978 raft_consensus.cc:385] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.848930 20978 raft_consensus.cc:740] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.849607 20978 consensus_queue.cc:260] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.857811 20978 ts_tablet_manager.cc:1434] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20260924 19:58:46.850108 20977 raft_consensus.cc:359] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.858155 20977 raft_consensus.cc:385] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.858937 20977 raft_consensus.cc:740] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.859700 20977 consensus_queue.cc:260] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.866700 20977 ts_tablet_manager.cc:1434] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.019s user 0.004s sys 0.002s
I20260924 19:58:46.867511 20977 tablet_bootstrap.cc:492] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:46.872877 20980 ts_tablet_manager.cc:1916] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:46.893673 20977 tablet_bootstrap.cc:654] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.897710 20978 tablet_bootstrap.cc:492] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:46.903043 20978 tablet_bootstrap.cc:654] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
W20260924 19:58:46.907184 21033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:58:46.916096 20978 tablet_bootstrap.cc:492] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:46.916180 20980 ts_tablet_manager.cc:1929] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:46.916498 20978 ts_tablet_manager.cc:1403] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20260924 19:58:46.916625 20980 log.cc:1199] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/3647da617a934fb69e70c056a7a88f47
I20260924 19:58:46.917312 20980 ts_tablet_manager.cc:1950] T 3647da617a934fb69e70c056a7a88f47 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:46.918998 20978 raft_consensus.cc:359] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.921325 20977 tablet_bootstrap.cc:492] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:46.921730 20977 ts_tablet_manager.cc:1403] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.002s
I20260924 19:58:46.923988 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 3647da617a934fb69e70c056a7a88f47 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:46.924100 20977 raft_consensus.cc:359] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.924716 20977 raft_consensus.cc:385] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.925932 20977 raft_consensus.cc:740] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.926541 20977 consensus_queue.cc:260] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.928871 20977 ts_tablet_manager.cc:1434] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 19:58:46.933638 20279 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c40348984314595b6e6e49c4f03f35e from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:46.919639 20978 raft_consensus.cc:385] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:46.934506 20978 raft_consensus.cc:740] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:46.935189 20978 consensus_queue.cc:260] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:46.952814 20978 ts_tablet_manager.cc:1434] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20260924 19:58:46.953832 20553 tablet_service.cc:1511] Processing CreateTablet for tablet 0c7d4336285d4ed2b8252609894f9a49 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.954941 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c7d4336285d4ed2b8252609894f9a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.976111 20702 tablet_service.cc:1511] Processing CreateTablet for tablet 0c7d4336285d4ed2b8252609894f9a49 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:46.977250 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c7d4336285d4ed2b8252609894f9a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:46.979604 20977 tablet_bootstrap.cc:492] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:46.982018 20976 tablet_bootstrap.cc:492] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:46.985370 20977 tablet_bootstrap.cc:654] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:46.986769 20976 tablet_bootstrap.cc:654] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:47.005955 20848 tablet_service.cc:1511] Processing CreateTablet for tablet 0c7d4336285d4ed2b8252609894f9a49 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:47.006690 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 1a12482cbc69483d8789f530de3393e1 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:47.007259 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c7d4336285d4ed2b8252609894f9a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:47.007496 20977 tablet_bootstrap.cc:492] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:47.007943 20977 ts_tablet_manager.cc:1403] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.004s
I20260924 19:58:47.010360 20977 raft_consensus.cc:359] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.012428 20976 tablet_bootstrap.cc:492] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:47.012750 20977 raft_consensus.cc:385] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.012822 20976 ts_tablet_manager.cc:1403] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.000s
I20260924 19:58:47.015247 20977 raft_consensus.cc:740] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.015297 20976 raft_consensus.cc:359] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.016142 20977 consensus_queue.cc:260] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.015887 20976 raft_consensus.cc:385] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.020303 20976 raft_consensus.cc:740] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.023411 20976 consensus_queue.cc:260] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.025251 20976 ts_tablet_manager.cc:1434] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260924 19:58:47.038764 20276 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 1a12482cbc69483d8789f530de3393e1
I20260924 19:58:47.040163 20977 ts_tablet_manager.cc:1434] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.032s user 0.004s sys 0.001s
I20260924 19:58:47.042183 20701 tablet_service.cc:1558] Processing DeleteTablet for tablet 1a12482cbc69483d8789f530de3393e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6767a95369834bd5b9084b4e614e5980) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:47.040659 20553 tablet_service.cc:1558] Processing DeleteTablet for tablet 1a12482cbc69483d8789f530de3393e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6767a95369834bd5b9084b4e614e5980) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:47.042951 20981 tablet_replica.cc:333] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:47.043087 20980 tablet_replica.cc:333] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:47.043486 20276 catalog_manager.cc:6934] ReplaceTablet: tablet 1a12482cbc69483d8789f530de3393e1 deleted and replaced by tablet 6767a95369834bd5b9084b4e614e5980
I20260924 19:58:47.043718 20981 raft_consensus.cc:2243] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.043834 20980 raft_consensus.cc:2243] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.044183 20981 raft_consensus.cc:2272] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.044282 20980 raft_consensus.cc:2272] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.046452 20981 ts_tablet_manager.cc:1916] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.049011 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 1a12482cbc69483d8789f530de3393e1 with delete_type TABLET_DATA_DELETED (replaced by tablet 6767a95369834bd5b9084b4e614e5980) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:47.049746 20982 tablet_replica.cc:333] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:47.050513 20982 raft_consensus.cc:2243] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.050990 20982 raft_consensus.cc:2272] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.056167 20982 ts_tablet_manager.cc:1916] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.058372 20980 ts_tablet_manager.cc:1916] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.090401 20982 ts_tablet_manager.cc:1929] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:47.090783 20982 log.cc:1199] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/1a12482cbc69483d8789f530de3393e1
I20260924 19:58:47.092059 20982 ts_tablet_manager.cc:1950] T 1a12482cbc69483d8789f530de3393e1 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:47.094502 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 1a12482cbc69483d8789f530de3393e1 (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:47.097100 20981 ts_tablet_manager.cc:1929] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:47.097492 20981 log.cc:1199] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/1a12482cbc69483d8789f530de3393e1
I20260924 19:58:47.098150 20981 ts_tablet_manager.cc:1950] T 1a12482cbc69483d8789f530de3393e1 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:47.100853 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 1a12482cbc69483d8789f530de3393e1 (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:47.105006 20980 ts_tablet_manager.cc:1929] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:47.105443 20980 log.cc:1199] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/1a12482cbc69483d8789f530de3393e1
I20260924 19:58:47.107723 20980 ts_tablet_manager.cc:1950] T 1a12482cbc69483d8789f530de3393e1 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:47.110849 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 1a12482cbc69483d8789f530de3393e1 (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:47.143893 20279 master_service.cc:960] ReplaceTablet: received request to replace tablet d26c86130a67479d9f8ef3a829f8f6a8 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:47.158495 20978 tablet_bootstrap.cc:492] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:47.171254 20978 tablet_bootstrap.cc:654] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:47.174434 20279 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet d26c86130a67479d9f8ef3a829f8f6a8
I20260924 19:58:47.176223 20279 catalog_manager.cc:6934] ReplaceTablet: tablet d26c86130a67479d9f8ef3a829f8f6a8 deleted and replaced by tablet d658e4b929c54aee9914d114fc1cd302
I20260924 19:58:47.177610 20553 tablet_service.cc:1558] Processing DeleteTablet for tablet d26c86130a67479d9f8ef3a829f8f6a8 with delete_type TABLET_DATA_DELETED (replaced by tablet d658e4b929c54aee9914d114fc1cd302) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:47.178393 20980 tablet_replica.cc:333] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:47.179451 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet d26c86130a67479d9f8ef3a829f8f6a8 with delete_type TABLET_DATA_DELETED (replaced by tablet d658e4b929c54aee9914d114fc1cd302) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:47.180162 20982 tablet_replica.cc:333] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:47.180795 20982 raft_consensus.cc:2243] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.181239 20982 raft_consensus.cc:2272] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.182848 20982 ts_tablet_manager.cc:1916] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.183059 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet d26c86130a67479d9f8ef3a829f8f6a8 with delete_type TABLET_DATA_DELETED (replaced by tablet d658e4b929c54aee9914d114fc1cd302) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:47.183183 20980 raft_consensus.cc:2243] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.183683 20980 raft_consensus.cc:2272] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.183825 20981 tablet_replica.cc:333] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:47.184494 20981 raft_consensus.cc:2243] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.184967 20981 raft_consensus.cc:2272] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.189034 20980 ts_tablet_manager.cc:1916] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.193744 20981 ts_tablet_manager.cc:1916] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.205658 20553 tablet_service.cc:1511] Processing CreateTablet for tablet 6767a95369834bd5b9084b4e614e5980 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:47.206926 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6767a95369834bd5b9084b4e614e5980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:47.212446 20701 tablet_service.cc:1511] Processing CreateTablet for tablet 6767a95369834bd5b9084b4e614e5980 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:47.213745 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6767a95369834bd5b9084b4e614e5980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:47.218252 20846 tablet_service.cc:1511] Processing CreateTablet for tablet 6767a95369834bd5b9084b4e614e5980 (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:47.219435 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6767a95369834bd5b9084b4e614e5980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:47.224642 20978 tablet_bootstrap.cc:492] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:47.225467 20978 ts_tablet_manager.cc:1403] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.000s
I20260924 19:58:47.227862 20978 raft_consensus.cc:359] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.230376 20978 raft_consensus.cc:385] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.230785 20978 raft_consensus.cc:740] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.231796 20997 raft_consensus.cc:493] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:47.232299 20997 raft_consensus.cc:515] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.234602 20997 leader_election.cc:290] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:47.237582 20980 ts_tablet_manager.cc:1929] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260924 19:58:47.238063 20508 leader_election.cc:343] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c7d4336285d4ed2b8252609894f9a49
I20260924 19:58:47.238090 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c7d4336285d4ed2b8252609894f9a49" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:47.238279 20980 log.cc:1199] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/d26c86130a67479d9f8ef3a829f8f6a8
I20260924 19:58:47.239171 20980 ts_tablet_manager.cc:1950] T d26c86130a67479d9f8ef3a829f8f6a8 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:47.244468 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet d26c86130a67479d9f8ef3a829f8f6a8 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:47.245900 20982 ts_tablet_manager.cc:1929] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:47.246264 20982 log.cc:1199] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/d26c86130a67479d9f8ef3a829f8f6a8
I20260924 19:58:47.246901 20982 ts_tablet_manager.cc:1950] T d26c86130a67479d9f8ef3a829f8f6a8 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:47.247511 20977 tablet_bootstrap.cc:492] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:47.247300 20978 consensus_queue.cc:260] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.248265 20722 raft_consensus.cc:2468] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 0.
I20260924 19:58:47.252732 20509 leader_election.cc:304] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c
I20260924 19:58:47.256081 20997 raft_consensus.cc:2804] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:47.256423 20997 raft_consensus.cc:493] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:47.256711 20997 raft_consensus.cc:3060] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:47.268018 20978 ts_tablet_manager.cc:1434] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.042s user 0.008s sys 0.000s
I20260924 19:58:47.268033 20275 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c40348984314595b6e6e49c4f03f35e from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:47.269199 20976 tablet_bootstrap.cc:492] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:47.271106 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet d26c86130a67479d9f8ef3a829f8f6a8 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:47.276512 20977 tablet_bootstrap.cc:654] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:47.284228 20555 tablet_service.cc:1511] Processing CreateTablet for tablet d658e4b929c54aee9914d114fc1cd302 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:47.285274 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d658e4b929c54aee9914d114fc1cd302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:47.280268 20997 raft_consensus.cc:515] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.290537 20702 tablet_service.cc:1511] Processing CreateTablet for tablet d658e4b929c54aee9914d114fc1cd302 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:47.291638 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d658e4b929c54aee9914d114fc1cd302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:47.292332 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c7d4336285d4ed2b8252609894f9a49" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
I20260924 19:58:47.292843 20722 raft_consensus.cc:3060] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:47.300807 20978 tablet_bootstrap.cc:492] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:47.304141 20981 ts_tablet_manager.cc:1929] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:47.304553 20981 log.cc:1199] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/d26c86130a67479d9f8ef3a829f8f6a8
I20260924 19:58:47.305147 20981 ts_tablet_manager.cc:1950] T d26c86130a67479d9f8ef3a829f8f6a8 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:47.306074 20978 tablet_bootstrap.cc:654] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:47.307713 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet d26c86130a67479d9f8ef3a829f8f6a8 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:47.309445 20976 tablet_bootstrap.cc:654] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:47.310626 20977 tablet_bootstrap.cc:492] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:47.311892 20977 ts_tablet_manager.cc:1403] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.065s user 0.011s sys 0.000s
I20260924 19:58:47.314457 20977 raft_consensus.cc:359] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.317159 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c7d4336285d4ed2b8252609894f9a49" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:47.321465 20722 raft_consensus.cc:2468] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 1.
I20260924 19:58:47.325186 20977 raft_consensus.cc:385] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.328598 20977 raft_consensus.cc:740] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
W20260924 19:58:47.318719 20508 leader_election.cc:343] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:47.315855 20997 leader_election.cc:290] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Requested vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:47.330106 20977 consensus_queue.cc:260] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.332084 20977 ts_tablet_manager.cc:1434] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.020s user 0.004s sys 0.002s
I20260924 19:58:47.348115 20846 tablet_service.cc:1511] Processing CreateTablet for tablet d658e4b929c54aee9914d114fc1cd302 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:47.349634 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d658e4b929c54aee9914d114fc1cd302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:47.340046 20509 leader_election.cc:304] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c
I20260924 19:58:47.351574 20997 raft_consensus.cc:2804] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:47.361173 20997 raft_consensus.cc:697] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: Becoming Leader. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Running, Role: LEADER
I20260924 19:58:47.362171 20997 consensus_queue.cc:237] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.386210 20976 tablet_bootstrap.cc:492] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:47.386694 20976 ts_tablet_manager.cc:1403] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.118s user 0.008s sys 0.004s
I20260924 19:58:47.388880 20976 raft_consensus.cc:359] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.389431 20976 raft_consensus.cc:385] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.389668 20976 raft_consensus.cc:740] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.390405 20976 consensus_queue.cc:260] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.401605 20976 ts_tablet_manager.cc:1434] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.015s user 0.000s sys 0.006s
I20260924 19:58:47.405531 20978 tablet_bootstrap.cc:492] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:47.406064 20978 ts_tablet_manager.cc:1403] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.105s user 0.023s sys 0.019s
I20260924 19:58:47.410566 20978 raft_consensus.cc:359] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.411728 20977 tablet_bootstrap.cc:492] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:47.411216 20978 raft_consensus.cc:385] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.412691 20978 raft_consensus.cc:740] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.416692 20977 tablet_bootstrap.cc:654] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:47.416992 20978 consensus_queue.cc:260] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.420224 20978 ts_tablet_manager.cc:1434] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260924 19:58:47.420997 20978 tablet_bootstrap.cc:492] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:47.435869 20978 tablet_bootstrap.cc:654] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:47.442955 21029 consensus_queue.cc:1048] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:58:47.452294 20977 tablet_bootstrap.cc:492] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:47.452718 20977 ts_tablet_manager.cc:1403] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.000s
I20260924 19:58:47.455451 20977 raft_consensus.cc:359] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.456079 20977 raft_consensus.cc:385] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.456352 20977 raft_consensus.cc:740] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.456970 20977 consensus_queue.cc:260] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.461894 20976 tablet_bootstrap.cc:492] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:47.462769 20978 tablet_bootstrap.cc:492] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:47.463194 20978 ts_tablet_manager.cc:1403] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.000s
I20260924 19:58:47.465682 20978 raft_consensus.cc:359] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.466465 20978 raft_consensus.cc:385] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.468981 20977 ts_tablet_manager.cc:1434] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260924 19:58:47.466722 20978 raft_consensus.cc:740] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.472361 20978 consensus_queue.cc:260] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.476946 20978 ts_tablet_manager.cc:1434] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.013s user 0.004s sys 0.001s
I20260924 19:58:47.508373 20279 catalog_manager.cc:5838] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "30a4f50fc28c4616ba622ba51b0b2622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20260924 19:58:47.518741 20275 master_service.cc:960] ReplaceTablet: received request to replace tablet d658e4b929c54aee9914d114fc1cd302 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:47.522542 20976 tablet_bootstrap.cc:654] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:47.528228 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet d658e4b929c54aee9914d114fc1cd302 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:47.569713 21029 consensus_queue.cc:1048] T 635f4961769f49dbb908af7904d29042 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:58:47.572304 20275 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet d658e4b929c54aee9914d114fc1cd302
I20260924 19:58:47.574277 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet d658e4b929c54aee9914d114fc1cd302 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a65531fe4cc4576a62753c0271baf16) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:47.575067 20980 tablet_replica.cc:333] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:47.575793 20980 raft_consensus.cc:2243] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.576259 20980 raft_consensus.cc:2272] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.578799 20980 ts_tablet_manager.cc:1916] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.588881 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet d658e4b929c54aee9914d114fc1cd302 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a65531fe4cc4576a62753c0271baf16) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:47.589712 20981 tablet_replica.cc:333] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:47.590221 20275 catalog_manager.cc:6934] ReplaceTablet: tablet d658e4b929c54aee9914d114fc1cd302 deleted and replaced by tablet 3a65531fe4cc4576a62753c0271baf16
I20260924 19:58:47.590360 20981 raft_consensus.cc:2243] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.590317 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet d658e4b929c54aee9914d114fc1cd302 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a65531fe4cc4576a62753c0271baf16) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:47.590806 20981 raft_consensus.cc:2272] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
W20260924 19:58:47.592190 20270 catalog_manager.cc:5030] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): delete failed for tablet d658e4b929c54aee9914d114fc1cd302 because the tablet was not found. No further retry: Not found: Tablet not found: d658e4b929c54aee9914d114fc1cd302
I20260924 19:58:47.593940 20981 ts_tablet_manager.cc:1916] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.621079 20976 tablet_bootstrap.cc:492] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:47.621534 20976 ts_tablet_manager.cc:1403] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.160s user 0.010s sys 0.000s
I20260924 19:58:47.623979 20976 raft_consensus.cc:359] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.624595 20976 raft_consensus.cc:385] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.624866 20976 raft_consensus.cc:740] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.628439 20981 ts_tablet_manager.cc:1929] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:47.628883 20981 log.cc:1199] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/d658e4b929c54aee9914d114fc1cd302
I20260924 19:58:47.629544 20981 ts_tablet_manager.cc:1950] T d658e4b929c54aee9914d114fc1cd302 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
W20260924 19:58:47.640657 20279 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:47.646090 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet d658e4b929c54aee9914d114fc1cd302 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
W20260924 19:58:47.655045 20279 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
W20260924 19:58:47.658522 20279 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:47.659962 20555 tablet_service.cc:1511] Processing CreateTablet for tablet 3a65531fe4cc4576a62753c0271baf16 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:47.661051 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a65531fe4cc4576a62753c0271baf16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:47.675453 20976 consensus_queue.cc:260] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:58:47.686519 20275 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:47.693671 20702 tablet_service.cc:1511] Processing CreateTablet for tablet 3a65531fe4cc4576a62753c0271baf16 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:47.694743 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a65531fe4cc4576a62753c0271baf16. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:58:47.699337 20275 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:47.708863 20980 ts_tablet_manager.cc:1929] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:47.709246 20980 log.cc:1199] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/d658e4b929c54aee9914d114fc1cd302
I20260924 19:58:47.709909 20980 ts_tablet_manager.cc:1950] T d658e4b929c54aee9914d114fc1cd302 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:47.713424 20976 ts_tablet_manager.cc:1434] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.092s user 0.005s sys 0.000s
I20260924 19:58:47.716962 20846 tablet_service.cc:1511] Processing CreateTablet for tablet 3a65531fe4cc4576a62753c0271baf16 (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:47.718102 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a65531fe4cc4576a62753c0271baf16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:47.732085 20978 tablet_bootstrap.cc:492] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:47.738641 20978 tablet_bootstrap.cc:654] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:47.744720 20976 tablet_bootstrap.cc:492] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:47.750458 20976 tablet_bootstrap.cc:654] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:47.752799 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet d658e4b929c54aee9914d114fc1cd302 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:47.756208 20977 tablet_bootstrap.cc:492] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:47.765024 20978 tablet_bootstrap.cc:492] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:47.765574 20978 ts_tablet_manager.cc:1403] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.008s
I20260924 19:58:47.766950 20977 tablet_bootstrap.cc:654] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
W20260924 19:58:47.777083 20270 consensus_peers.cc:597] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 -> Peer 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311): Couldn't send request to peer 022c1898fcbe4856aa680c26047f6934. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260924 19:58:47.777665 20978 raft_consensus.cc:359] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.778543 21058 raft_consensus.cc:493] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:47.778985 21058 raft_consensus.cc:515] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.780884 21058 leader_election.cc:290] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:47.781930 20275 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c7d4336285d4ed2b8252609894f9a49 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:47.785892 20977 tablet_bootstrap.cc:492] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:47.786300 20977 ts_tablet_manager.cc:1403] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.000s
I20260924 19:58:47.788051 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6767a95369834bd5b9084b4e614e5980" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:47.788626 20575 raft_consensus.cc:2468] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
I20260924 19:58:47.788748 20977 raft_consensus.cc:359] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.789430 20977 raft_consensus.cc:385] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.789682 20977 raft_consensus.cc:740] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.790211 20804 leader_election.cc:304] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters:
I20260924 19:58:47.791122 21058 raft_consensus.cc:2804] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:47.783084 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6767a95369834bd5b9084b4e614e5980" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:47.791438 21058 raft_consensus.cc:493] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:47.791750 21058 raft_consensus.cc:3060] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:47.791946 20722 raft_consensus.cc:2468] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
I20260924 19:58:47.794735 20978 raft_consensus.cc:385] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.795140 20978 raft_consensus.cc:740] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.795867 20977 consensus_queue.cc:260] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.795868 20978 consensus_queue.cc:260] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.797937 20978 ts_tablet_manager.cc:1434] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260924 19:58:47.805094 20977 ts_tablet_manager.cc:1434] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20260924 19:58:47.808826 20976 tablet_bootstrap.cc:492] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:47.809355 20976 ts_tablet_manager.cc:1403] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.003s
I20260924 19:58:47.812108 20976 raft_consensus.cc:359] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.813025 20976 raft_consensus.cc:385] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.813361 20976 raft_consensus.cc:740] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.814276 20976 consensus_queue.cc:260] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.818109 20976 ts_tablet_manager.cc:1434] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260924 19:58:47.833272 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet d658e4b929c54aee9914d114fc1cd302 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a65531fe4cc4576a62753c0271baf16) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:47.833976 20982 tablet_replica.cc:333] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:47.834643 20982 raft_consensus.cc:2243] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.835057 20982 raft_consensus.cc:2272] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.837486 20982 ts_tablet_manager.cc:1916] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.838192 20976 tablet_bootstrap.cc:492] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:47.854233 20976 tablet_bootstrap.cc:654] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:47.855578 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 3a65531fe4cc4576a62753c0271baf16 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:47.857650 20275 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 0c7d4336285d4ed2b8252609894f9a49
I20260924 19:58:47.859586 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c7d4336285d4ed2b8252609894f9a49 with delete_type TABLET_DATA_DELETED (replaced by tablet 1021c8766ff74481abf4a7aafddcb5a7) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:47.860529 20980 tablet_replica.cc:333] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:47.860792 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c7d4336285d4ed2b8252609894f9a49 with delete_type TABLET_DATA_DELETED (replaced by tablet 1021c8766ff74481abf4a7aafddcb5a7) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:47.861466 20980 raft_consensus.cc:2243] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:47.862318 20275 catalog_manager.cc:6934] ReplaceTablet: tablet 0c7d4336285d4ed2b8252609894f9a49 deleted and replaced by tablet 1021c8766ff74481abf4a7aafddcb5a7
I20260924 19:58:47.862416 20980 pending_rounds.cc:70] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: Trying to abort 1 pending ops.
I20260924 19:58:47.863888 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 0c7d4336285d4ed2b8252609894f9a49 with delete_type TABLET_DATA_DELETED (replaced by tablet 1021c8766ff74481abf4a7aafddcb5a7) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:47.864028 21019 raft_consensus.cc:493] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:47.862746 20980 pending_rounds.cc:77] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986582648995840 op_type: NO_OP noop_request { }
I20260924 19:58:47.864616 20980 raft_consensus.cc:2889] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:58:47.864441 21019 raft_consensus.cc:515] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.865033 20980 raft_consensus.cc:2272] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.866268 21019 leader_election.cc:290] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:47.868083 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a65531fe4cc4576a62753c0271baf16" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:47.868626 20575 raft_consensus.cc:2468] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:47.869833 20658 leader_election.cc:304] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:47.870621 21019 raft_consensus.cc:2804] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:47.870774 20980 ts_tablet_manager.cc:1916] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.870937 21019 raft_consensus.cc:493] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:47.871241 21019 raft_consensus.cc:3060] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:47.876345 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a65531fe4cc4576a62753c0271baf16" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
W20260924 19:58:47.877841 20657 leader_election.cc:343] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:47.887483 20982 ts_tablet_manager.cc:1929] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:47.887877 20982 log.cc:1199] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/d658e4b929c54aee9914d114fc1cd302
I20260924 19:58:47.888522 20982 ts_tablet_manager.cc:1950] T d658e4b929c54aee9914d114fc1cd302 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:47.893239 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet d658e4b929c54aee9914d114fc1cd302 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:47.894274 21058 raft_consensus.cc:515] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.899605 20981 tablet_replica.cc:333] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:47.901495 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6767a95369834bd5b9084b4e614e5980" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:47.902035 20575 raft_consensus.cc:3060] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:47.906279 20982 tablet_replica.cc:333] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:47.907223 20982 raft_consensus.cc:2243] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.907773 20982 raft_consensus.cc:2272] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.908152 20276 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 3a65531fe4cc4576a62753c0271baf16
I20260924 19:58:47.910027 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet 3a65531fe4cc4576a62753c0271baf16 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c1fdba2289b4ce89d8764f5799b5a3f) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:47.910252 20981 raft_consensus.cc:2243] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.910748 20981 raft_consensus.cc:2272] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.912140 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 3a65531fe4cc4576a62753c0271baf16 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c1fdba2289b4ce89d8764f5799b5a3f) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:47.912719 20981 ts_tablet_manager.cc:1916] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.914199 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 3a65531fe4cc4576a62753c0271baf16 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c1fdba2289b4ce89d8764f5799b5a3f) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:47.914824 20276 catalog_manager.cc:6934] ReplaceTablet: tablet 3a65531fe4cc4576a62753c0271baf16 deleted and replaced by tablet 6c1fdba2289b4ce89d8764f5799b5a3f
I20260924 19:58:47.917461 20976 tablet_bootstrap.cc:492] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:47.917838 20976 ts_tablet_manager.cc:1403] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.080s user 0.025s sys 0.016s
I20260924 19:58:47.920266 20976 raft_consensus.cc:359] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.920882 20976 raft_consensus.cc:385] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:47.921141 20976 raft_consensus.cc:740] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:47.921834 20976 consensus_queue.cc:260] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.922755 20982 ts_tablet_manager.cc:1916] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.925176 20575 raft_consensus.cc:2468] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 1.
W20260924 19:58:47.926136 20436 hms_client.cc:355] Time spent drop HMS table: real 1.204s user 0.000s sys 0.000s
I20260924 19:58:47.927141 20278 catalog_manager.cc:2770] deleted HMS catalog entry for table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]
I20260924 19:58:47.928395 20804 leader_election.cc:304] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters:
I20260924 19:58:47.929347 21060 raft_consensus.cc:2804] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:47.929785 21060 raft_consensus.cc:697] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: Becoming Leader. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Running, Role: LEADER
I20260924 19:58:47.930534 21060 consensus_queue.cc:237] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.934732 20980 ts_tablet_manager.cc:1929] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:47.935094 20980 log.cc:1199] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/0c7d4336285d4ed2b8252609894f9a49
I20260924 19:58:47.935837 20980 ts_tablet_manager.cc:1950] T 0c7d4336285d4ed2b8252609894f9a49 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:47.937584 20976 ts_tablet_manager.cc:1434] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20260924 19:58:47.938138 20980 tablet_replica.cc:333] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:47.938407 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 0c7d4336285d4ed2b8252609894f9a49 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:47.938795 20980 raft_consensus.cc:2243] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.940850 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6767a95369834bd5b9084b4e614e5980" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
I20260924 19:58:47.941392 20722 raft_consensus.cc:3060] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:47.947732 20722 raft_consensus.cc:2468] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 1.
I20260924 19:58:47.950248 20981 ts_tablet_manager.cc:1929] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:47.950744 20981 log.cc:1199] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/0c7d4336285d4ed2b8252609894f9a49
I20260924 19:58:47.951613 20981 ts_tablet_manager.cc:1950] T 0c7d4336285d4ed2b8252609894f9a49 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:47.952039 21058 leader_election.cc:290] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:47.956154 20981 tablet_replica.cc:333] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:47.956264 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 0c7d4336285d4ed2b8252609894f9a49 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:47.922480 21019 raft_consensus.cc:515] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:47.958663 20981 raft_consensus.cc:2243] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:47.960794 20981 raft_consensus.cc:2272] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.963339 20980 raft_consensus.cc:2272] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:47.964665 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a65531fe4cc4576a62753c0271baf16" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
W20260924 19:58:47.966153 20658 leader_election.cc:343] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:47.967378 20980 ts_tablet_manager.cc:1916] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.969249 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a65531fe4cc4576a62753c0271baf16" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:47.969816 20867 raft_consensus.cc:3060] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:47.975086 20982 ts_tablet_manager.cc:1929] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:47.976543 20867 raft_consensus.cc:2468] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:47.977770 20657 leader_election.cc:304] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters: 30a4f50fc28c4616ba622ba51b0b2622
W20260924 19:58:47.978448 20657 raft_consensus.cc:2686] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260924 19:58:47.979583 21019 leader_election.cc:290] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:47.982147 20981 ts_tablet_manager.cc:1916] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:47.998464 20982 log.cc:1199] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/0c7d4336285d4ed2b8252609894f9a49
I20260924 19:58:47.999565 20982 ts_tablet_manager.cc:1950] T 0c7d4336285d4ed2b8252609894f9a49 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:48.010172 20555 tablet_service.cc:1511] Processing CreateTablet for tablet 1021c8766ff74481abf4a7aafddcb5a7 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.011468 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1021c8766ff74481abf4a7aafddcb5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.021975 20982 tablet_replica.cc:333] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:48.022653 20982 raft_consensus.cc:2243] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.023206 20982 raft_consensus.cc:2272] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.028249 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 0c7d4336285d4ed2b8252609894f9a49 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:48.030440 20977 tablet_bootstrap.cc:492] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:48.032481 20982 ts_tablet_manager.cc:1916] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.032032 20275 catalog_manager.cc:5838] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0055218dbcbd48b58018e9774e59e14c (127.17.2.131). New cstate: current_term: 1 leader_uuid: "0055218dbcbd48b58018e9774e59e14c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:48.036187 20977 tablet_bootstrap.cc:654] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.052328 20977 tablet_bootstrap.cc:492] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:48.052855 20977 ts_tablet_manager.cc:1403] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.002s
I20260924 19:58:48.054971 20982 ts_tablet_manager.cc:1929] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.055341 20982 log.cc:1199] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/3a65531fe4cc4576a62753c0271baf16
I20260924 19:58:48.055940 20982 ts_tablet_manager.cc:1950] T 3a65531fe4cc4576a62753c0271baf16 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:48.055747 20977 raft_consensus.cc:359] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.057207 20977 raft_consensus.cc:385] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.057870 20977 raft_consensus.cc:740] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.059281 20977 consensus_queue.cc:260] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.039023 20981 ts_tablet_manager.cc:1929] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.062016 20981 log.cc:1199] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/3a65531fe4cc4576a62753c0271baf16
I20260924 19:58:48.062668 20981 ts_tablet_manager.cc:1950] T 3a65531fe4cc4576a62753c0271baf16 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:48.068236 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 3a65531fe4cc4576a62753c0271baf16 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:48.068783 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 3a65531fe4cc4576a62753c0271baf16 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:48.072261 20980 ts_tablet_manager.cc:1929] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.074931 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c40348984314595b6e6e49c4f03f35e from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:48.075299 20980 log.cc:1199] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/3a65531fe4cc4576a62753c0271baf16
I20260924 19:58:48.075371 20702 tablet_service.cc:1511] Processing CreateTablet for tablet 1021c8766ff74481abf4a7aafddcb5a7 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.076335 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1021c8766ff74481abf4a7aafddcb5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.076282 20980 ts_tablet_manager.cc:1950] T 3a65531fe4cc4576a62753c0271baf16 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:48.077977 20977 ts_tablet_manager.cc:1434] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260924 19:58:48.079506 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 3a65531fe4cc4576a62753c0271baf16 (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:48.094333 20846 tablet_service.cc:1511] Processing CreateTablet for tablet 1021c8766ff74481abf4a7aafddcb5a7 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.096020 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1021c8766ff74481abf4a7aafddcb5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.121697 20997 raft_consensus.cc:493] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:48.122151 20997 raft_consensus.cc:515] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.124200 20997 leader_election.cc:290] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:48.125875 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1021c8766ff74481abf4a7aafddcb5a7" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
W20260924 19:58:48.127328 20508 leader_election.cc:343] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 1021c8766ff74481abf4a7aafddcb5a7
W20260924 19:58:48.127800 20509 leader_election.cc:343] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:48.128187 20509 leader_election.cc:304] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:58:48.129168 20997 raft_consensus.cc:2749] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:48.129702 20978 tablet_bootstrap.cc:492] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:48.130997 20976 tablet_bootstrap.cc:492] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:48.134994 20978 tablet_bootstrap.cc:654] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.139410 20976 tablet_bootstrap.cc:654] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.152218 20976 tablet_bootstrap.cc:492] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:48.152812 20976 ts_tablet_manager.cc:1403] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.007s
I20260924 19:58:48.156272 20976 raft_consensus.cc:359] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.158087 20275 master_service.cc:960] ReplaceTablet: received request to replace tablet 6767a95369834bd5b9084b4e614e5980 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:48.162551 20976 raft_consensus.cc:385] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.162956 20976 raft_consensus.cc:740] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.165391 20976 consensus_queue.cc:260] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.185575 20976 ts_tablet_manager.cc:1434] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.032s user 0.009s sys 0.000s
I20260924 19:58:48.193495 20275 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 6767a95369834bd5b9084b4e614e5980
I20260924 19:58:48.195298 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet 6767a95369834bd5b9084b4e614e5980 with delete_type TABLET_DATA_DELETED (replaced by tablet d0787be6ea144c21ae179f69cb5f89ab) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:48.196101 20980 tablet_replica.cc:333] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:48.196637 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 6767a95369834bd5b9084b4e614e5980 with delete_type TABLET_DATA_DELETED (replaced by tablet d0787be6ea144c21ae179f69cb5f89ab) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:48.197031 20980 raft_consensus.cc:2243] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.197608 20980 raft_consensus.cc:2272] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.197824 20275 catalog_manager.cc:6934] ReplaceTablet: tablet 6767a95369834bd5b9084b4e614e5980 deleted and replaced by tablet d0787be6ea144c21ae179f69cb5f89ab
I20260924 19:58:48.197947 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 6767a95369834bd5b9084b4e614e5980 with delete_type TABLET_DATA_DELETED (replaced by tablet d0787be6ea144c21ae179f69cb5f89ab) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:48.198621 20982 tablet_replica.cc:333] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:48.199239 20982 raft_consensus.cc:2243] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:48.199326 20980 ts_tablet_manager.cc:1916] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.199982 20982 pending_rounds.cc:70] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: Trying to abort 1 pending ops.
I20260924 19:58:48.200273 20982 pending_rounds.cc:77] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986584814497792 op_type: NO_OP noop_request { }
I20260924 19:58:48.200618 20982 raft_consensus.cc:2889] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:58:48.200951 20982 raft_consensus.cc:2272] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.203301 20982 ts_tablet_manager.cc:1916] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.203454 20981 tablet_replica.cc:333] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:48.204110 20981 raft_consensus.cc:2243] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.204540 20981 raft_consensus.cc:2272] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.206214 20981 ts_tablet_manager.cc:1916] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.206372 20555 tablet_service.cc:1511] Processing CreateTablet for tablet 6c1fdba2289b4ce89d8764f5799b5a3f (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.207460 20978 tablet_bootstrap.cc:492] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:48.207460 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c1fdba2289b4ce89d8764f5799b5a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.207834 20978 ts_tablet_manager.cc:1403] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.078s user 0.010s sys 0.000s
I20260924 19:58:48.210129 20978 raft_consensus.cc:359] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.210721 20978 raft_consensus.cc:385] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.210958 20978 raft_consensus.cc:740] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.214931 20702 tablet_service.cc:1511] Processing CreateTablet for tablet 6c1fdba2289b4ce89d8764f5799b5a3f (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.214761 20978 consensus_queue.cc:260] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.219476 20846 tablet_service.cc:1511] Processing CreateTablet for tablet 6c1fdba2289b4ce89d8764f5799b5a3f (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.220518 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c1fdba2289b4ce89d8764f5799b5a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.215848 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c1fdba2289b4ce89d8764f5799b5a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.224390 20978 ts_tablet_manager.cc:1434] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260924 19:58:48.251953 20981 ts_tablet_manager.cc:1929] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.252175 20976 tablet_bootstrap.cc:492] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:48.252331 20981 log.cc:1199] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/6767a95369834bd5b9084b4e614e5980
I20260924 19:58:48.252940 20981 ts_tablet_manager.cc:1950] T 6767a95369834bd5b9084b4e614e5980 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:48.255473 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 6767a95369834bd5b9084b4e614e5980 (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:48.257395 20980 ts_tablet_manager.cc:1929] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.257709 20976 tablet_bootstrap.cc:654] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.257777 20980 log.cc:1199] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/6767a95369834bd5b9084b4e614e5980
I20260924 19:58:48.258391 20980 ts_tablet_manager.cc:1950] T 6767a95369834bd5b9084b4e614e5980 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:48.260988 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 6767a95369834bd5b9084b4e614e5980 (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:48.264629 20976 tablet_bootstrap.cc:492] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:48.264988 20976 ts_tablet_manager.cc:1403] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260924 19:58:48.266739 20553 tablet_service.cc:1511] Processing CreateTablet for tablet d0787be6ea144c21ae179f69cb5f89ab (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.267849 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0787be6ea144c21ae179f69cb5f89ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.269532 20982 ts_tablet_manager.cc:1929] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:48.270051 20982 log.cc:1199] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/6767a95369834bd5b9084b4e614e5980
I20260924 19:58:48.282874 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 1021c8766ff74481abf4a7aafddcb5a7 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:48.275624 20976 raft_consensus.cc:359] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.290431 20976 raft_consensus.cc:385] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.290840 20976 raft_consensus.cc:740] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.271046 20982 ts_tablet_manager.cc:1950] T 6767a95369834bd5b9084b4e614e5980 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:48.299068 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 6767a95369834bd5b9084b4e614e5980 (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:48.299939 20976 consensus_queue.cc:260] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.302760 20977 tablet_bootstrap.cc:492] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:48.308527 20977 tablet_bootstrap.cc:654] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.311625 20276 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 1021c8766ff74481abf4a7aafddcb5a7
I20260924 19:58:48.313475 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet 1021c8766ff74481abf4a7aafddcb5a7 with delete_type TABLET_DATA_DELETED (replaced by tablet a98b566994844f44983d40a00c5bcda3) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:48.314195 20980 tablet_replica.cc:333] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:48.314872 20980 raft_consensus.cc:2243] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.315320 20980 raft_consensus.cc:2272] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.315722 20701 tablet_service.cc:1511] Processing CreateTablet for tablet d0787be6ea144c21ae179f69cb5f89ab (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.316782 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0787be6ea144c21ae179f69cb5f89ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.316716 20846 tablet_service.cc:1511] Processing CreateTablet for tablet d0787be6ea144c21ae179f69cb5f89ab (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.317412 20980 ts_tablet_manager.cc:1916] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.317904 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0787be6ea144c21ae179f69cb5f89ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.332372 20976 ts_tablet_manager.cc:1434] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.067s user 0.007s sys 0.000s
I20260924 19:58:48.336390 20700 tablet_service.cc:1558] Processing DeleteTablet for tablet 1021c8766ff74481abf4a7aafddcb5a7 with delete_type TABLET_DATA_DELETED (replaced by tablet a98b566994844f44983d40a00c5bcda3) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:48.336870 20276 catalog_manager.cc:6934] ReplaceTablet: tablet 1021c8766ff74481abf4a7aafddcb5a7 deleted and replaced by tablet a98b566994844f44983d40a00c5bcda3
I20260924 19:58:48.337175 20981 tablet_replica.cc:333] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:48.337815 20981 raft_consensus.cc:2243] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.338249 20981 raft_consensus.cc:2272] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.340149 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet 1021c8766ff74481abf4a7aafddcb5a7 with delete_type TABLET_DATA_DELETED (replaced by tablet a98b566994844f44983d40a00c5bcda3) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:48.340317 20981 ts_tablet_manager.cc:1916] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.341636 20982 tablet_replica.cc:333] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:48.342254 20982 raft_consensus.cc:2243] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.342679 20982 raft_consensus.cc:2272] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.345860 20982 ts_tablet_manager.cc:1916] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.350442 21060 raft_consensus.cc:493] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:48.351125 21060 raft_consensus.cc:515] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.353080 21060 leader_election.cc:290] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:48.354789 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c1fdba2289b4ce89d8764f5799b5a3f" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
W20260924 19:58:48.357407 20804 leader_election.cc:343] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 6c1fdba2289b4ce89d8764f5799b5a3f
W20260924 19:58:48.358127 20804 leader_election.cc:343] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:48.358510 20804 leader_election.cc:304] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:58:48.359736 21060 raft_consensus.cc:2749] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:48.362128 20980 ts_tablet_manager.cc:1929] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.362480 20980 log.cc:1199] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/1021c8766ff74481abf4a7aafddcb5a7
I20260924 19:58:48.369308 20978 tablet_bootstrap.cc:492] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:48.372002 20982 ts_tablet_manager.cc:1929] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.372350 20982 log.cc:1199] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/1021c8766ff74481abf4a7aafddcb5a7
I20260924 19:58:48.372946 20982 ts_tablet_manager.cc:1950] T 1021c8766ff74481abf4a7aafddcb5a7 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:48.375624 20978 tablet_bootstrap.cc:654] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.375924 20981 ts_tablet_manager.cc:1929] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.376286 20981 log.cc:1199] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/1021c8766ff74481abf4a7aafddcb5a7
I20260924 19:58:48.376577 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 1021c8766ff74481abf4a7aafddcb5a7 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:48.376920 20981 ts_tablet_manager.cc:1950] T 1021c8766ff74481abf4a7aafddcb5a7 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:48.379405 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 1021c8766ff74481abf4a7aafddcb5a7 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:48.363137 20980 ts_tablet_manager.cc:1950] T 1021c8766ff74481abf4a7aafddcb5a7 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:48.379920 20977 tablet_bootstrap.cc:492] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:48.381711 20977 ts_tablet_manager.cc:1403] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.000s
I20260924 19:58:48.383524 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 1021c8766ff74481abf4a7aafddcb5a7 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:48.385387 20977 raft_consensus.cc:359] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.386107 20977 raft_consensus.cc:385] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.386186 20700 tablet_service.cc:1511] Processing CreateTablet for tablet a98b566994844f44983d40a00c5bcda3 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.386417 20977 raft_consensus.cc:740] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.387369 20700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98b566994844f44983d40a00c5bcda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.388228 20555 tablet_service.cc:1511] Processing CreateTablet for tablet a98b566994844f44983d40a00c5bcda3 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.389246 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98b566994844f44983d40a00c5bcda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.390621 20976 tablet_bootstrap.cc:492] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:48.396373 20976 tablet_bootstrap.cc:654] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.401834 20848 tablet_service.cc:1511] Processing CreateTablet for tablet a98b566994844f44983d40a00c5bcda3 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.403008 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98b566994844f44983d40a00c5bcda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.406189 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 6c1fdba2289b4ce89d8764f5799b5a3f from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:48.387317 20977 consensus_queue.cc:260] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.432717 20276 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 6c1fdba2289b4ce89d8764f5799b5a3f
I20260924 19:58:48.435704 20276 catalog_manager.cc:6934] ReplaceTablet: tablet 6c1fdba2289b4ce89d8764f5799b5a3f deleted and replaced by tablet d3797e87218f42f9bbb4cd717214e8fb
I20260924 19:58:48.436671 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c1fdba2289b4ce89d8764f5799b5a3f with delete_type TABLET_DATA_DELETED (replaced by tablet d3797e87218f42f9bbb4cd717214e8fb) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:48.437369 20982 tablet_replica.cc:333] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:48.438292 20982 raft_consensus.cc:2243] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.438943 20982 raft_consensus.cc:2272] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.440523 20553 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c1fdba2289b4ce89d8764f5799b5a3f with delete_type TABLET_DATA_DELETED (replaced by tablet d3797e87218f42f9bbb4cd717214e8fb) from {username='slave'} at 127.0.0.1:48342
W20260924 19:58:48.442104 20271 catalog_manager.cc:5043] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet 6c1fdba2289b4ce89d8764f5799b5a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c1fdba2289b4ce89d8764f5799b5a3f already in progress: creating tablet
I20260924 19:58:48.442390 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c1fdba2289b4ce89d8764f5799b5a3f with delete_type TABLET_DATA_DELETED (replaced by tablet d3797e87218f42f9bbb4cd717214e8fb) from {username='slave'} at 127.0.0.1:60830
W20260924 19:58:48.444180 20271 catalog_manager.cc:5043] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet 6c1fdba2289b4ce89d8764f5799b5a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c1fdba2289b4ce89d8764f5799b5a3f already in progress: creating tablet
I20260924 19:58:48.444690 20978 tablet_bootstrap.cc:492] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:48.445109 20978 ts_tablet_manager.cc:1403] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.000s
I20260924 19:58:48.445600 20982 ts_tablet_manager.cc:1916] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.447490 20978 raft_consensus.cc:359] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.448156 20978 raft_consensus.cc:385] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.448432 20978 raft_consensus.cc:740] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.449165 20978 consensus_queue.cc:260] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.465785 20553 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c1fdba2289b4ce89d8764f5799b5a3f with delete_type TABLET_DATA_DELETED (replaced by tablet d3797e87218f42f9bbb4cd717214e8fb) from {username='slave'} at 127.0.0.1:48342
W20260924 19:58:48.467432 20271 catalog_manager.cc:5043] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet 6c1fdba2289b4ce89d8764f5799b5a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c1fdba2289b4ce89d8764f5799b5a3f already in progress: creating tablet
I20260924 19:58:48.473130 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c1fdba2289b4ce89d8764f5799b5a3f with delete_type TABLET_DATA_DELETED (replaced by tablet d3797e87218f42f9bbb4cd717214e8fb) from {username='slave'} at 127.0.0.1:60830
W20260924 19:58:48.474750 20271 catalog_manager.cc:5043] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet 6c1fdba2289b4ce89d8764f5799b5a3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c1fdba2289b4ce89d8764f5799b5a3f already in progress: creating tablet
I20260924 19:58:48.476020 20976 tablet_bootstrap.cc:492] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:48.476414 20976 ts_tablet_manager.cc:1403] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.086s user 0.019s sys 0.012s
I20260924 19:58:48.476197 20977 ts_tablet_manager.cc:1434] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.094s user 0.005s sys 0.000s
I20260924 19:58:48.478663 20976 raft_consensus.cc:359] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.480718 20976 raft_consensus.cc:385] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.480979 20976 raft_consensus.cc:740] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.482810 20976 consensus_queue.cc:260] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.485687 20976 ts_tablet_manager.cc:1434] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260924 19:58:48.486500 20976 tablet_bootstrap.cc:492] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:48.486840 20977 tablet_bootstrap.cc:492] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:48.489691 20978 ts_tablet_manager.cc:1434] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.044s user 0.005s sys 0.000s
I20260924 19:58:48.491959 20553 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c1fdba2289b4ce89d8764f5799b5a3f with delete_type TABLET_DATA_DELETED (replaced by tablet d3797e87218f42f9bbb4cd717214e8fb) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:48.492707 20980 tablet_replica.cc:333] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:48.493348 20980 raft_consensus.cc:2243] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.493784 20980 raft_consensus.cc:2272] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.495946 20980 ts_tablet_manager.cc:1916] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.502315 20976 tablet_bootstrap.cc:654] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.503360 20982 ts_tablet_manager.cc:1929] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.503824 20982 log.cc:1199] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/6c1fdba2289b4ce89d8764f5799b5a3f
I20260924 19:58:48.504802 20982 ts_tablet_manager.cc:1950] T 6c1fdba2289b4ce89d8764f5799b5a3f P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:48.516076 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 6c1fdba2289b4ce89d8764f5799b5a3f (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:48.517814 20701 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c1fdba2289b4ce89d8764f5799b5a3f with delete_type TABLET_DATA_DELETED (replaced by tablet d3797e87218f42f9bbb4cd717214e8fb) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:48.518493 20981 tablet_replica.cc:333] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:48.519119 20981 raft_consensus.cc:2243] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.519549 20981 raft_consensus.cc:2272] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.521190 20976 tablet_bootstrap.cc:492] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:48.521579 20976 ts_tablet_manager.cc:1403] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.003s
I20260924 19:58:48.522702 20981 ts_tablet_manager.cc:1916] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.523861 20976 raft_consensus.cc:359] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.524428 20976 raft_consensus.cc:385] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.524688 20976 raft_consensus.cc:740] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.531430 20977 tablet_bootstrap.cc:654] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.532888 20976 consensus_queue.cc:260] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.541442 20978 tablet_bootstrap.cc:492] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:48.542080 20976 ts_tablet_manager.cc:1434] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260924 19:58:48.544881 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet d0787be6ea144c21ae179f69cb5f89ab from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:48.546890 20978 tablet_bootstrap.cc:654] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.559198 20553 tablet_service.cc:1511] Processing CreateTablet for tablet d3797e87218f42f9bbb4cd717214e8fb (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.563484 20700 tablet_service.cc:1511] Processing CreateTablet for tablet d3797e87218f42f9bbb4cd717214e8fb (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.564576 20700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3797e87218f42f9bbb4cd717214e8fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.572743 20980 ts_tablet_manager.cc:1929] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.574283 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3797e87218f42f9bbb4cd717214e8fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.573808 20980 log.cc:1199] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/6c1fdba2289b4ce89d8764f5799b5a3f
I20260924 19:58:48.579991 20980 ts_tablet_manager.cc:1950] T 6c1fdba2289b4ce89d8764f5799b5a3f P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:48.582458 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 6c1fdba2289b4ce89d8764f5799b5a3f (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:48.595266 20276 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet d0787be6ea144c21ae179f69cb5f89ab
I20260924 19:58:48.597860 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet d0787be6ea144c21ae179f69cb5f89ab with delete_type TABLET_DATA_DELETED (replaced by tablet a8d34fa77ecb4102919ff5fe6fd8412e) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:48.599397 20701 tablet_service.cc:1558] Processing DeleteTablet for tablet d0787be6ea144c21ae179f69cb5f89ab with delete_type TABLET_DATA_DELETED (replaced by tablet a8d34fa77ecb4102919ff5fe6fd8412e) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:48.600281 20276 catalog_manager.cc:6934] ReplaceTablet: tablet d0787be6ea144c21ae179f69cb5f89ab deleted and replaced by tablet a8d34fa77ecb4102919ff5fe6fd8412e
W20260924 19:58:48.602496 20271 catalog_manager.cc:5043] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet d0787be6ea144c21ae179f69cb5f89ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0787be6ea144c21ae179f69cb5f89ab already in progress: creating tablet
I20260924 19:58:48.602540 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet d0787be6ea144c21ae179f69cb5f89ab with delete_type TABLET_DATA_DELETED (replaced by tablet a8d34fa77ecb4102919ff5fe6fd8412e) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:48.603314 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet d0787be6ea144c21ae179f69cb5f89ab with delete_type TABLET_DATA_DELETED (replaced by tablet a8d34fa77ecb4102919ff5fe6fd8412e) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:48.603552 20982 tablet_replica.cc:333] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:48.604578 20982 raft_consensus.cc:2243] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.605221 20982 raft_consensus.cc:2272] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.608106 20982 ts_tablet_manager.cc:1916] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.623680 20848 tablet_service.cc:1511] Processing CreateTablet for tablet d3797e87218f42f9bbb4cd717214e8fb (DEFAULT_TABLE table=default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.624816 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3797e87218f42f9bbb4cd717214e8fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.639113 20976 tablet_bootstrap.cc:492] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:48.646414 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet d0787be6ea144c21ae179f69cb5f89ab from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:48.649083 20976 tablet_bootstrap.cc:654] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.653785 20280 catalog_manager.cc:2213] created HMS catalog entry for table [id=6539df5762ac4b4f9663be322513b2a0]
I20260924 19:58:48.654951 20982 ts_tablet_manager.cc:1929] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.655484 20982 log.cc:1199] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/d0787be6ea144c21ae179f69cb5f89ab
I20260924 19:58:48.656370 20982 ts_tablet_manager.cc:1950] T d0787be6ea144c21ae179f69cb5f89ab P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:48.661227 20978 tablet_bootstrap.cc:492] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:48.664419 20976 tablet_bootstrap.cc:492] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:48.664611 20978 ts_tablet_manager.cc:1403] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.123s user 0.005s sys 0.004s
I20260924 19:58:48.665041 20976 ts_tablet_manager.cc:1403] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.009s
I20260924 19:58:48.668953 20978 raft_consensus.cc:359] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.669587 20978 raft_consensus.cc:385] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.669492 20280 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 20) took 3236 ms. Trace:
I20260924 19:58:48.669822 20978 raft_consensus.cc:740] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.671960 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet d0787be6ea144c21ae179f69cb5f89ab (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:48.675657 20976 raft_consensus.cc:359] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.676617 20976 raft_consensus.cc:385] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.676982 20976 raft_consensus.cc:740] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.677883 20976 consensus_queue.cc:260] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.669854 20280 rpcz_store.cc:276] 0924 19:58:45.432811 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:45.432928 (+ 117us) service_pool.cc:224] Handling call
0924 19:58:45.553522 (+120594us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:45.558225 (+ 4703us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:48.653737 (+3095512us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:48.668788 (+ 15051us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:58:48.669124 (+ 336us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:58:48.669413 (+ 289us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:48.655721 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:48.656181 (+ 460us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:48.656209 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:58:48.656226 (+ 17us) tablet.cc:662] Decoding operations
0924 19:58:48.656542 (+ 316us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:48.656588 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:48.656613 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:48.656839 (+ 226us) tablet.cc:701] Row locks acquired
0924 19:58:48.656852 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:58:48.656986 (+ 134us) write_op.cc:270] Start()
0924 19:58:48.657078 (+ 92us) write_op.cc:276] Timestamp: P: 1790279928656912 usec, L: 0
0924 19:58:48.657091 (+ 13us) op_driver.cc:348] REPLICATION: starting
0924 19:58:48.657634 (+ 543us) log.cc:844] Serialized 432 byte log entry
0924 19:58:48.666055 (+ 8421us) op_driver.cc:464] REPLICATION: finished
0924 19:58:48.666513 (+ 458us) write_op.cc:301] APPLY: starting
0924 19:58:48.666569 (+ 56us) tablet.cc:1394] starting BulkCheckPresence
0924 19:58:48.666675 (+ 106us) tablet.cc:1397] finished BulkCheckPresence
0924 19:58:48.666761 (+ 86us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:58:48.667136 (+ 375us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:58:48.667292 (+ 156us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:58:48.667335 (+ 43us) write_op.cc:312] APPLY: finished
0924 19:58:48.667525 (+ 190us) log.cc:844] Serialized 28 byte log entry
0924 19:58:48.667656 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:48.667725 (+ 69us) write_op.cc:454] Released schema lock
0924 19:58:48.667757 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0924 19:58:48.667974 (+ 217us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:48.667997 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":2367734,"Hive Metastore.run_cpu_time_us":662,"Hive Metastore.run_wall_time_us":726398,"child_traces":[["op",{"apply.queue_time_us":286,"apply.run_cpu_time_us":1432,"apply.run_wall_time_us":1431,"mutex_wait_us":159,"num_ops":2,"prepare.queue_time_us":182,"prepare.run_cpu_time_us":2316,"prepare.run_wall_time_us":2463,"raft.queue_time_us":439,"raft.run_cpu_time_us":729,"raft.run_wall_time_us":728,"replication_time_us":8861}]]}
W20260924 19:58:48.680759 20270 catalog_manager.cc:5030] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): delete failed for tablet d0787be6ea144c21ae179f69cb5f89ab because the tablet was not found. No further retry: Not found: Tablet not found: d0787be6ea144c21ae179f69cb5f89ab
I20260924 19:58:48.682354 20276 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_08eddf6c85b94591a21958b5f18ada13" } modify_external_catalogs: true
I20260924 19:58:48.683126 20276 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_08eddf6c85b94591a21958b5f18ada13" } modify_external_catalogs: true
I20260924 19:58:48.692725 20977 tablet_bootstrap.cc:492] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:48.693164 20977 ts_tablet_manager.cc:1403] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.214s user 0.019s sys 0.027s
I20260924 19:58:48.699710 20977 raft_consensus.cc:359] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.700676 20976 ts_tablet_manager.cc:1434] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.035s user 0.009s sys 0.000s
I20260924 19:58:48.700338 20977 raft_consensus.cc:385] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.701350 20977 raft_consensus.cc:740] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.702147 20977 consensus_queue.cc:260] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.704795 20981 ts_tablet_manager.cc:1929] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.705195 20981 log.cc:1199] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/6c1fdba2289b4ce89d8764f5799b5a3f
I20260924 19:58:48.705839 20981 ts_tablet_manager.cc:1950] T 6c1fdba2289b4ce89d8764f5799b5a3f P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:48.707830 20977 ts_tablet_manager.cc:1434] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260924 19:58:48.708676 20977 tablet_bootstrap.cc:492] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:48.712177 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet d0787be6ea144c21ae179f69cb5f89ab with delete_type TABLET_DATA_DELETED (replaced by tablet a8d34fa77ecb4102919ff5fe6fd8412e) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:48.718461 20980 tablet_replica.cc:333] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:48.719081 20980 raft_consensus.cc:2243] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.719496 20980 raft_consensus.cc:2272] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.723665 20980 ts_tablet_manager.cc:1916] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.670588 20978 consensus_queue.cc:260] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.745402 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 6c1fdba2289b4ce89d8764f5799b5a3f (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
W20260924 19:58:48.747077 20271 catalog_manager.cc:5043] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet d0787be6ea144c21ae179f69cb5f89ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0787be6ea144c21ae179f69cb5f89ab already in progress: creating tablet
I20260924 19:58:48.743017 20978 ts_tablet_manager.cc:1434] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.078s user 0.005s sys 0.000s
I20260924 19:58:48.763841 20701 tablet_service.cc:1558] Processing DeleteTablet for tablet d0787be6ea144c21ae179f69cb5f89ab with delete_type TABLET_DATA_DELETED (replaced by tablet a8d34fa77ecb4102919ff5fe6fd8412e) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:48.765829 20978 tablet_bootstrap.cc:492] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:48.771293 20981 tablet_replica.cc:333] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:48.772046 20981 raft_consensus.cc:2243] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.773454 20981 raft_consensus.cc:2272] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.799374 20701 tablet_service.cc:1511] Processing CreateTablet for tablet a8d34fa77ecb4102919ff5fe6fd8412e (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.800472 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8d34fa77ecb4102919ff5fe6fd8412e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:58:48.804046 20280 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:48.815073 20977 tablet_bootstrap.cc:654] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.818231 20280 master_service.cc:960] ReplaceTablet: received request to replace tablet a98b566994844f44983d40a00c5bcda3 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:48.823189 20978 tablet_bootstrap.cc:654] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.826239 20980 ts_tablet_manager.cc:1929] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.826601 20980 log.cc:1199] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/d0787be6ea144c21ae179f69cb5f89ab
I20260924 19:58:48.827232 20980 ts_tablet_manager.cc:1950] T d0787be6ea144c21ae179f69cb5f89ab P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:48.830384 20553 tablet_service.cc:1511] Processing CreateTablet for tablet a8d34fa77ecb4102919ff5fe6fd8412e (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.832296 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet d0787be6ea144c21ae179f69cb5f89ab (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:48.843627 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8d34fa77ecb4102919ff5fe6fd8412e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.849995 20977 tablet_bootstrap.cc:492] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:48.850407 20977 ts_tablet_manager.cc:1403] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.142s user 0.003s sys 0.006s
I20260924 19:58:48.856014 20280 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet a98b566994844f44983d40a00c5bcda3
I20260924 19:58:48.869575 20848 tablet_service.cc:1511] Processing CreateTablet for tablet a8d34fa77ecb4102919ff5fe6fd8412e (DEFAULT_TABLE table=default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:48.869809 20700 tablet_service.cc:1558] Processing DeleteTablet for tablet a98b566994844f44983d40a00c5bcda3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fa9ceda714341e5b8eeb4144fea1342) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:48.870645 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8d34fa77ecb4102919ff5fe6fd8412e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:48.873842 20977 raft_consensus.cc:359] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.874504 20977 raft_consensus.cc:385] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.874770 20977 raft_consensus.cc:740] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.876866 20978 tablet_bootstrap.cc:492] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:48.878074 20978 ts_tablet_manager.cc:1403] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.112s user 0.010s sys 0.000s
I20260924 19:58:48.880559 20978 raft_consensus.cc:359] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.881179 20978 raft_consensus.cc:385] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:48.881433 20978 raft_consensus.cc:740] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:48.882056 20978 consensus_queue.cc:260] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.885887 20279 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c40348984314595b6e6e49c4f03f35e from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:48.888213 20275 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_7a523607282c4d1c85c7a42deabb470b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:48.895562 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet a98b566994844f44983d40a00c5bcda3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fa9ceda714341e5b8eeb4144fea1342) from {username='slave'} at 127.0.0.1:48342
W20260924 19:58:48.897332 20271 catalog_manager.cc:5043] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet a98b566994844f44983d40a00c5bcda3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a98b566994844f44983d40a00c5bcda3 already in progress: creating tablet
I20260924 19:58:48.899586 20280 catalog_manager.cc:6934] ReplaceTablet: tablet a98b566994844f44983d40a00c5bcda3 deleted and replaced by tablet 5fa9ceda714341e5b8eeb4144fea1342
I20260924 19:58:48.899814 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet a98b566994844f44983d40a00c5bcda3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fa9ceda714341e5b8eeb4144fea1342) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:48.900852 20982 tablet_replica.cc:333] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:48.800475 20981 ts_tablet_manager.cc:1916] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.901800 20982 raft_consensus.cc:2243] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.902388 20982 raft_consensus.cc:2272] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.903467 20977 consensus_queue.cc:260] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:48.908161 20978 ts_tablet_manager.cc:1434] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20260924 19:58:48.909044 20978 tablet_bootstrap.cc:492] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:48.917153 20977 ts_tablet_manager.cc:1434] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.066s user 0.006s sys 0.001s
I20260924 19:58:48.921146 20982 ts_tablet_manager.cc:1916] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.922241 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet a98b566994844f44983d40a00c5bcda3 with delete_type TABLET_DATA_DELETED (replaced by tablet 5fa9ceda714341e5b8eeb4144fea1342) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:48.923116 20980 tablet_replica.cc:333] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:48.923840 20980 raft_consensus.cc:2243] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:48.924324 20980 raft_consensus.cc:2272] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:48.927423 20980 ts_tablet_manager.cc:1916] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:48.962622 20977 tablet_bootstrap.cc:492] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:48.968617 20976 tablet_bootstrap.cc:492] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:48.977159 20976 tablet_bootstrap.cc:654] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.987449 20976 tablet_bootstrap.cc:492] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:48.989984 20977 tablet_bootstrap.cc:654] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.989058 20982 ts_tablet_manager.cc:1929] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:48.990692 20982 log.cc:1199] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/a98b566994844f44983d40a00c5bcda3
I20260924 19:58:48.990017 20976 ts_tablet_manager.cc:1403] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.000s
I20260924 19:58:48.996755 20978 tablet_bootstrap.cc:654] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:48.997757 20982 ts_tablet_manager.cc:1950] T a98b566994844f44983d40a00c5bcda3 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:49.002046 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet a98b566994844f44983d40a00c5bcda3 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:48.996830 20976 raft_consensus.cc:359] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.007846 20976 raft_consensus.cc:385] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.008795 20976 raft_consensus.cc:740] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.010857 20976 consensus_queue.cc:260] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.030292 20976 ts_tablet_manager.cc:1434] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s
I20260924 19:58:49.048434 20978 tablet_bootstrap.cc:492] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:49.049522 20978 ts_tablet_manager.cc:1403] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.141s user 0.010s sys 0.000s
I20260924 19:58:49.051916 20848 tablet_service.cc:1511] Processing CreateTablet for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 (DEFAULT_TABLE table=default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.052934 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b925a15a6074be0ae5d2c43a7ab83d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.052858 20701 tablet_service.cc:1511] Processing CreateTablet for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 (DEFAULT_TABLE table=default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.053900 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b925a15a6074be0ae5d2c43a7ab83d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.058391 20553 tablet_service.cc:1511] Processing CreateTablet for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 (DEFAULT_TABLE table=default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.059495 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b925a15a6074be0ae5d2c43a7ab83d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.067772 20981 ts_tablet_manager.cc:1929] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:49.068153 20981 log.cc:1199] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/d0787be6ea144c21ae179f69cb5f89ab
I20260924 19:58:49.068778 20981 ts_tablet_manager.cc:1950] T d0787be6ea144c21ae179f69cb5f89ab P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:49.069954 20555 tablet_service.cc:1511] Processing CreateTablet for tablet 5fa9ceda714341e5b8eeb4144fea1342 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.070897 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fa9ceda714341e5b8eeb4144fea1342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.071858 20978 raft_consensus.cc:359] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.072464 20978 raft_consensus.cc:385] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.072703 20978 raft_consensus.cc:740] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.074268 21019 raft_consensus.cc:493] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:49.074699 21019 raft_consensus.cc:515] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.076579 21019 leader_election.cc:290] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:49.079676 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a98b566994844f44983d40a00c5bcda3" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
W20260924 19:58:49.081084 20657 leader_election.cc:343] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: a98b566994844f44983d40a00c5bcda3
W20260924 19:58:49.082036 20658 leader_election.cc:343] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Illegal state: Tablet not RUNNING: STOPPED
I20260924 19:58:49.082402 20658 leader_election.cc:304] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:58:49.083328 20980 ts_tablet_manager.cc:1929] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:49.083675 20980 log.cc:1199] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/a98b566994844f44983d40a00c5bcda3
I20260924 19:58:49.084298 20980 ts_tablet_manager.cc:1950] T a98b566994844f44983d40a00c5bcda3 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:49.086584 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet a98b566994844f44983d40a00c5bcda3 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:49.087330 21019 raft_consensus.cc:2749] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:49.093537 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet d0787be6ea144c21ae179f69cb5f89ab (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:49.095105 20981 tablet_replica.cc:333] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:49.095778 20981 raft_consensus.cc:2243] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:49.096238 20981 raft_consensus.cc:2272] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:49.099510 20280 master_service.cc:960] ReplaceTablet: received request to replace tablet d3797e87218f42f9bbb4cd717214e8fb from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:49.099486 20978 consensus_queue.cc:260] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.108738 20279 master_service.cc:960] ReplaceTablet: received request to replace tablet d3797e87218f42f9bbb4cd717214e8fb from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:49.109332 20978 ts_tablet_manager.cc:1434] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.060s user 0.005s sys 0.000s
I20260924 19:58:49.117856 20978 tablet_bootstrap.cc:492] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:49.118304 20976 tablet_bootstrap.cc:492] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:49.124153 20700 tablet_service.cc:1511] Processing CreateTablet for tablet 5fa9ceda714341e5b8eeb4144fea1342 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.125227 20700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fa9ceda714341e5b8eeb4144fea1342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.112910 20981 ts_tablet_manager.cc:1916] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:49.135485 20976 tablet_bootstrap.cc:654] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:49.149142 20976 tablet_bootstrap.cc:492] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:49.151564 20976 ts_tablet_manager.cc:1403] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.005s
I20260924 19:58:49.155537 20976 raft_consensus.cc:359] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.160872 20976 raft_consensus.cc:385] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.161857 20976 raft_consensus.cc:740] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.165886 20978 tablet_bootstrap.cc:654] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:49.166661 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet d3797e87218f42f9bbb4cd717214e8fb with delete_type TABLET_DATA_DELETED (replaced by tablet 62f4d38a8da14252982bcac0e7df28c4) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:49.165936 20976 consensus_queue.cc:260] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.179250 20280 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet d3797e87218f42f9bbb4cd717214e8fb
I20260924 19:58:49.180976 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet d3797e87218f42f9bbb4cd717214e8fb with delete_type TABLET_DATA_DELETED (replaced by tablet 62f4d38a8da14252982bcac0e7df28c4) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:49.181831 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet d3797e87218f42f9bbb4cd717214e8fb with delete_type TABLET_DATA_DELETED (replaced by tablet 62f4d38a8da14252982bcac0e7df28c4) from {username='slave'} at 127.0.0.1:60830
W20260924 19:58:49.182544 20271 catalog_manager.cc:5043] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet d3797e87218f42f9bbb4cd717214e8fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3797e87218f42f9bbb4cd717214e8fb already in progress: creating tablet
I20260924 19:58:49.183310 20280 catalog_manager.cc:6934] ReplaceTablet: tablet d3797e87218f42f9bbb4cd717214e8fb deleted and replaced by tablet 62f4d38a8da14252982bcac0e7df28c4
I20260924 19:58:49.185290 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet d3797e87218f42f9bbb4cd717214e8fb with delete_type TABLET_DATA_DELETED (replaced by tablet 62f4d38a8da14252982bcac0e7df28c4) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:49.186061 20982 tablet_replica.cc:333] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:49.186691 20982 raft_consensus.cc:2243] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:49.187100 20982 raft_consensus.cc:2272] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:49.188702 20982 ts_tablet_manager.cc:1916] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:49.199433 20284 catalog_manager.cc:2213] created HMS catalog entry for table [id=c2d09462b2b14d1a862ef2b18152c4e1]
I20260924 19:58:49.203831 20976 ts_tablet_manager.cc:1434] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.052s user 0.007s sys 0.000s
W20260924 19:58:49.230181 20279 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:49.232353 20284 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 17) took 4175 ms. Trace:
I20260924 19:58:49.232720 20284 rpcz_store.cc:276] 0924 19:58:45.057004 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:45.057137 (+ 133us) service_pool.cc:224] Handling call
0924 19:58:45.077118 (+ 19981us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b54000104d4. stack: 00007fe7ffab1323 00007fe7ffab1220 00007fe7ff655574 0000000000516964 00000000005168f9 00007fe806baf5c0 00007fe806c4e6fa 00007fe8089be98c 00007fe808abe166 00007fe80666a034 00007fe806669fc2 00007fe806669ef1 00007fe806669e6c 00007fe806669122 00007fe80491409c 00007fe8049134d6
0924 19:58:45.528858 (+451740us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:45.558919 (+ 30061us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:49.199376 (+3640457us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:49.231655 (+ 32279us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:58:49.231995 (+ 340us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:58:49.232287 (+ 292us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:49.201432 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:49.201805 (+ 373us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:49.201828 (+ 23us) write_op.cc:435] Acquired schema lock
0924 19:58:49.201841 (+ 13us) tablet.cc:662] Decoding operations
0924 19:58:49.202081 (+ 240us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:49.202114 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:49.202135 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:49.202316 (+ 181us) tablet.cc:701] Row locks acquired
0924 19:58:49.202329 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:58:49.202429 (+ 100us) write_op.cc:270] Start()
0924 19:58:49.202502 (+ 73us) write_op.cc:276] Timestamp: P: 1790279929202379 usec, L: 0
0924 19:58:49.202517 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:58:49.204290 (+ 1773us) log.cc:844] Serialized 432 byte log entry
0924 19:58:49.229061 (+ 24771us) op_driver.cc:464] REPLICATION: finished
0924 19:58:49.230236 (+ 1175us) write_op.cc:301] APPLY: starting
0924 19:58:49.230289 (+ 53us) tablet.cc:1394] starting BulkCheckPresence
0924 19:58:49.230389 (+ 100us) tablet.cc:1397] finished BulkCheckPresence
0924 19:58:49.230474 (+ 85us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:58:49.230839 (+ 365us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:58:49.230969 (+ 130us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:58:49.231010 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:58:49.231201 (+ 191us) log.cc:844] Serialized 28 byte log entry
0924 19:58:49.231325 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:49.231413 (+ 88us) write_op.cc:454] Released schema lock
0924 19:58:49.231443 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":3093688,"Hive Metastore.run_cpu_time_us":626,"Hive Metastore.run_wall_time_us":545861,"spinlock_wait_cycles":43894144,"child_traces":[["op",{"apply.queue_time_us":1034,"apply.run_cpu_time_us":1421,"apply.run_wall_time_us":5061,"num_ops":2,"prepare.queue_time_us":198,"prepare.run_cpu_time_us":1755,"prepare.run_wall_time_us":4126,"raft.queue_time_us":72,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":15,"replication_time_us":26451}]]}
I20260924 19:58:49.245446 20982 ts_tablet_manager.cc:1929] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:49.246629 20982 log.cc:1199] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/d3797e87218f42f9bbb4cd717214e8fb
I20260924 19:58:49.248242 20982 ts_tablet_manager.cc:1950] T d3797e87218f42f9bbb4cd717214e8fb P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:49.264911 20978 tablet_bootstrap.cc:492] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:49.265372 20978 ts_tablet_manager.cc:1403] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.005s
I20260924 19:58:49.272152 20848 tablet_service.cc:1511] Processing CreateTablet for tablet 5fa9ceda714341e5b8eeb4144fea1342 (DEFAULT_TABLE table=default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.273034 20978 raft_consensus.cc:359] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.273658 20978 raft_consensus.cc:385] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.273916 20978 raft_consensus.cc:740] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.274580 20978 consensus_queue.cc:260] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.278414 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fa9ceda714341e5b8eeb4144fea1342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.280798 20981 ts_tablet_manager.cc:1929] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:49.281143 20981 log.cc:1199] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/a98b566994844f44983d40a00c5bcda3
I20260924 19:58:49.281733 20981 ts_tablet_manager.cc:1950] T a98b566994844f44983d40a00c5bcda3 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:49.284264 20977 tablet_bootstrap.cc:492] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:49.284719 20977 ts_tablet_manager.cc:1403] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.334s user 0.007s sys 0.011s
I20260924 19:58:49.287005 20977 raft_consensus.cc:359] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.287722 20977 raft_consensus.cc:385] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.287967 20977 raft_consensus.cc:740] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.288558 20977 consensus_queue.cc:260] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.297010 20977 ts_tablet_manager.cc:1434] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20260924 19:58:49.304415 20977 tablet_bootstrap.cc:492] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:49.255030 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet d3797e87218f42f9bbb4cd717214e8fb (table default.table_11b41e77c9bf468c95a9cbe331cdf531 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
W20260924 19:58:49.320446 20270 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.17.2.131:41601, user_credentials={real_user=slave}} blocked reactor thread for 65412.7us
I20260924 19:58:49.320971 20978 ts_tablet_manager.cc:1434] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.055s user 0.007s sys 0.000s
I20260924 19:58:49.324002 20978 tablet_bootstrap.cc:492] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:49.336097 20977 tablet_bootstrap.cc:654] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
W20260924 19:58:49.344339 20438 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62f4d38a8da14252982bcac0e7df28c4 has no consensus state
I20260924 19:58:49.348294 20981 tablet_replica.cc:333] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:49.348948 20981 raft_consensus.cc:2243] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:49.349376 20981 raft_consensus.cc:2272] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:49.351750 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet a98b566994844f44983d40a00c5bcda3 (table default.table_9d6179f80e96408995081e5524a3422a [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:49.351325 20981 ts_tablet_manager.cc:1916] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:49.374469 20978 tablet_bootstrap.cc:654] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:49.379765 20280 master_service.cc:960] ReplaceTablet: received request to replace tablet d3797e87218f42f9bbb4cd717214e8fb from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:49.395413 20997 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:49.395907 20997 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.397847 20997 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:58:49.399662 20271 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 62f4d38a8da14252982bcac0e7df28c4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 62f4d38a8da14252982bcac0e7df28c4 has no consensus state
W20260924 19:58:49.401510 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 19:58:49.402514 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:58:49.402876 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:58:49.403075 20284 master_service.cc:960] ReplaceTablet: received request to replace tablet d3797e87218f42f9bbb4cd717214e8fb from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:49.403975 20997 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:49.406183 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet d3797e87218f42f9bbb4cd717214e8fb with delete_type TABLET_DATA_DELETED (replaced by tablet 62f4d38a8da14252982bcac0e7df28c4) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:49.406837 20980 tablet_replica.cc:333] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:49.407506 20980 raft_consensus.cc:2243] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:49.407922 20980 raft_consensus.cc:2272] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:49.408231 20977 tablet_bootstrap.cc:492] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:49.408690 20977 ts_tablet_manager.cc:1403] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.105s user 0.010s sys 0.000s
I20260924 19:58:49.409904 20980 ts_tablet_manager.cc:1916] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:49.411026 20977 raft_consensus.cc:359] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.411708 20977 raft_consensus.cc:385] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.412007 20977 raft_consensus.cc:740] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.412708 20977 consensus_queue.cc:260] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.431875 20553 tablet_service.cc:1511] Processing CreateTablet for tablet 8ebcdc9a748549f7bddb69bb5b99b315 (DEFAULT_TABLE table=default.table_14cbcbeb7f98417197bb13b1f8b5af49 [id=c2d09462b2b14d1a862ef2b18152c4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.433061 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ebcdc9a748549f7bddb69bb5b99b315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.441606 20977 ts_tablet_manager.cc:1434] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.033s user 0.004s sys 0.000s
I20260924 19:58:49.441771 20700 tablet_service.cc:1511] Processing CreateTablet for tablet 8ebcdc9a748549f7bddb69bb5b99b315 (DEFAULT_TABLE table=default.table_14cbcbeb7f98417197bb13b1f8b5af49 [id=c2d09462b2b14d1a862ef2b18152c4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.445176 20700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ebcdc9a748549f7bddb69bb5b99b315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.449822 20977 tablet_bootstrap.cc:492] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:49.452340 20284 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_2a7727ff525b44bc97232bb07f824105"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:49.459882 20846 tablet_service.cc:1511] Processing CreateTablet for tablet 8ebcdc9a748549f7bddb69bb5b99b315 (DEFAULT_TABLE table=default.table_14cbcbeb7f98417197bb13b1f8b5af49 [id=c2d09462b2b14d1a862ef2b18152c4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.460856 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ebcdc9a748549f7bddb69bb5b99b315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.463133 20701 tablet_service.cc:1511] Processing CreateTablet for tablet 62f4d38a8da14252982bcac0e7df28c4 (DEFAULT_TABLE table=default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.464177 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f4d38a8da14252982bcac0e7df28c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.460243 20555 tablet_service.cc:1511] Processing CreateTablet for tablet 62f4d38a8da14252982bcac0e7df28c4 (DEFAULT_TABLE table=default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.469269 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f4d38a8da14252982bcac0e7df28c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.468025 20997 raft_consensus.cc:493] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:49.470867 20997 raft_consensus.cc:515] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.474207 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8d34fa77ecb4102919ff5fe6fd8412e" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:49.474869 20867 raft_consensus.cc:2468] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 0.
I20260924 19:58:49.475988 20997 leader_election.cc:290] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:49.477180 20508 leader_election.cc:304] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters:
I20260924 19:58:49.477650 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8d34fa77ecb4102919ff5fe6fd8412e" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:49.478049 20997 raft_consensus.cc:2804] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:49.478363 20997 raft_consensus.cc:493] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:49.478272 20722 raft_consensus.cc:2468] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 0.
I20260924 19:58:49.478654 20997 raft_consensus.cc:3060] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:49.490564 20847 tablet_service.cc:1511] Processing CreateTablet for tablet 62f4d38a8da14252982bcac0e7df28c4 (DEFAULT_TABLE table=default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.492100 20847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f4d38a8da14252982bcac0e7df28c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.493360 20997 raft_consensus.cc:515] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.496188 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8d34fa77ecb4102919ff5fe6fd8412e" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
I20260924 19:58:49.496732 20722 raft_consensus.cc:3060] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:49.504240 20722 raft_consensus.cc:2468] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 1.
I20260924 19:58:49.505591 20509 leader_election.cc:304] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:49.511281 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a8d34fa77ecb4102919ff5fe6fd8412e" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:49.511863 20867 raft_consensus.cc:3060] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:49.512542 20976 tablet_bootstrap.cc:492] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:49.514998 20980 ts_tablet_manager.cc:1929] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:49.515400 20980 log.cc:1199] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/d3797e87218f42f9bbb4cd717214e8fb
I20260924 19:58:49.518013 20976 tablet_bootstrap.cc:654] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:49.518916 20867 raft_consensus.cc:2468] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 1.
I20260924 19:58:49.520150 21099 raft_consensus.cc:2804] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:49.520785 20978 tablet_bootstrap.cc:492] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:49.521248 20978 ts_tablet_manager.cc:1403] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.200s user 0.006s sys 0.005s
I20260924 19:58:49.523655 20978 raft_consensus.cc:359] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.524335 20978 raft_consensus.cc:385] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.524627 20978 raft_consensus.cc:740] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.526345 20978 consensus_queue.cc:260] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.533066 20997 leader_election.cc:290] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Requested vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:49.516049 20980 ts_tablet_manager.cc:1950] T d3797e87218f42f9bbb4cd717214e8fb P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:49.546502 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet d3797e87218f42f9bbb4cd717214e8fb (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:49.550055 20978 ts_tablet_manager.cc:1434] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20260924 19:58:49.559463 20978 tablet_bootstrap.cc:492] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:49.568693 20977 tablet_bootstrap.cc:654] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:49.568949 21086 raft_consensus.cc:493] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:49.569672 21086 raft_consensus.cc:515] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.574043 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b925a15a6074be0ae5d2c43a7ab83d1" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:49.576737 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b925a15a6074be0ae5d2c43a7ab83d1" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:49.577288 21086 leader_election.cc:290] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667)
I20260924 19:58:49.576789 20722 raft_consensus.cc:2468] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
W20260924 19:58:49.578766 20804 leader_election.cc:343] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:49.581025 20804 leader_election.cc:304] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters: 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:58:49.582785 21086 raft_consensus.cc:2804] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:49.583588 21086 raft_consensus.cc:493] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:49.584159 21086 raft_consensus.cc:3060] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:58:49.590314 20438 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5fa9ceda714341e5b8eeb4144fea1342 consensus state has no leader
I20260924 19:58:49.532745 20981 ts_tablet_manager.cc:1929] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:49.593050 20981 log.cc:1199] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/d3797e87218f42f9bbb4cd717214e8fb
I20260924 19:58:49.593798 20981 ts_tablet_manager.cc:1950] T d3797e87218f42f9bbb4cd717214e8fb P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:49.594652 20976 tablet_bootstrap.cc:492] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:49.595181 20976 ts_tablet_manager.cc:1403] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.083s user 0.012s sys 0.009s
I20260924 19:58:49.595216 21099 raft_consensus.cc:697] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: Becoming Leader. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Running, Role: LEADER
I20260924 19:58:49.596118 21099 consensus_queue.cc:237] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.597620 20976 raft_consensus.cc:359] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.598268 20976 raft_consensus.cc:385] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.598526 20976 raft_consensus.cc:740] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.601356 21086 raft_consensus.cc:515] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.606617 20978 tablet_bootstrap.cc:654] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:49.608723 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet d3797e87218f42f9bbb4cd717214e8fb (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:49.609288 20438 catalog_manager.cc:2811] Deleting table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1] in response to Hive Metastore notification log event 7
I20260924 19:58:49.609262 20976 consensus_queue.cc:260] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:58:49.610347 20271 catalog_manager.cc:5030] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet d3797e87218f42f9bbb4cd717214e8fb because the tablet was not found. No further retry: Not found: Tablet not found: d3797e87218f42f9bbb4cd717214e8fb
I20260924 19:58:49.617245 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b925a15a6074be0ae5d2c43a7ab83d1" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
I20260924 19:58:49.617853 20722 raft_consensus.cc:3060] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:49.627724 20722 raft_consensus.cc:2468] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 1.
I20260924 19:58:49.629889 20804 leader_election.cc:304] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:49.631059 20280 master_service.cc:960] ReplaceTablet: received request to replace tablet 5fa9ceda714341e5b8eeb4144fea1342 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:49.652272 21105 raft_consensus.cc:2804] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:49.654493 20978 tablet_bootstrap.cc:492] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:49.655009 20978 ts_tablet_manager.cc:1403] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.004s
I20260924 19:58:49.657727 20978 raft_consensus.cc:359] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.658389 20978 raft_consensus.cc:385] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.658660 20978 raft_consensus.cc:740] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.663745 21105 raft_consensus.cc:697] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: Becoming Leader. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Running, Role: LEADER
I20260924 19:58:49.672386 21108 raft_consensus.cc:493] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:49.658545 21086 leader_election.cc:290] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Requested vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667)
I20260924 19:58:49.674075 20976 ts_tablet_manager.cc:1434] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.079s user 0.005s sys 0.002s
I20260924 19:58:49.675396 21108 raft_consensus.cc:515] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.677508 21108 leader_election.cc:290] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:49.679461 20978 consensus_queue.cc:260] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.681360 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ebcdc9a748549f7bddb69bb5b99b315" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
W20260924 19:58:49.683245 20804 leader_election.cc:343] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:49.685352 20976 tablet_bootstrap.cc:492] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:49.685701 21105 consensus_queue.cc:237] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.648130 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b925a15a6074be0ae5d2c43a7ab83d1" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:49.678433 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ebcdc9a748549f7bddb69bb5b99b315" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
W20260924 19:58:49.692663 20804 leader_election.cc:343] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:49.693087 20804 leader_election.cc:304] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:58:49.693894 21108 raft_consensus.cc:2749] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:58:49.695250 20804 leader_election.cc:343] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:49.636631 20977 tablet_bootstrap.cc:492] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:49.697590 20977 ts_tablet_manager.cc:1403] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.248s user 0.010s sys 0.000s
I20260924 19:58:49.699749 20978 ts_tablet_manager.cc:1434] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.044s user 0.004s sys 0.001s
I20260924 19:58:49.701354 20976 tablet_bootstrap.cc:654] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:49.716933 20978 tablet_bootstrap.cc:492] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:49.730100 20280 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 5fa9ceda714341e5b8eeb4144fea1342
I20260924 19:58:49.731887 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fa9ceda714341e5b8eeb4144fea1342 with delete_type TABLET_DATA_DELETED (replaced by tablet 82554a94caa44720b56d1ee51dd11163) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:49.732618 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fa9ceda714341e5b8eeb4144fea1342 with delete_type TABLET_DATA_DELETED (replaced by tablet 82554a94caa44720b56d1ee51dd11163) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:49.734242 20700 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fa9ceda714341e5b8eeb4144fea1342 with delete_type TABLET_DATA_DELETED (replaced by tablet 82554a94caa44720b56d1ee51dd11163) from {username='slave'} at 127.0.0.1:60830
W20260924 19:58:49.734355 20270 catalog_manager.cc:5043] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): delete failed for tablet 5fa9ceda714341e5b8eeb4144fea1342 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fa9ceda714341e5b8eeb4144fea1342 already in progress: creating tablet
I20260924 19:58:49.734938 20280 catalog_manager.cc:6934] ReplaceTablet: tablet 5fa9ceda714341e5b8eeb4144fea1342 deleted and replaced by tablet 82554a94caa44720b56d1ee51dd11163
I20260924 19:58:49.734980 20981 tablet_replica.cc:333] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:49.735683 20981 raft_consensus.cc:2243] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:49.736125 20981 raft_consensus.cc:2272] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:49.736732 20976 tablet_bootstrap.cc:492] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
W20260924 19:58:49.737697 20271 catalog_manager.cc:5043] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet 5fa9ceda714341e5b8eeb4144fea1342 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5fa9ceda714341e5b8eeb4144fea1342 already in progress: creating tablet
I20260924 19:58:49.739409 20976 ts_tablet_manager.cc:1403] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.009s
I20260924 19:58:49.741801 20981 ts_tablet_manager.cc:1916] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:49.747789 20976 raft_consensus.cc:359] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.749166 20976 raft_consensus.cc:385] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.749464 20976 raft_consensus.cc:740] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.750146 20976 consensus_queue.cc:260] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.762344 20976 ts_tablet_manager.cc:1434] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.023s user 0.004s sys 0.002s
I20260924 19:58:49.762627 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet a8d34fa77ecb4102919ff5fe6fd8412e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:49 UTC) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:49.763341 20980 tablet_replica.cc:333] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:49.764021 20980 raft_consensus.cc:2243] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:49.764712 20980 pending_rounds.cc:70] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: Trying to abort 1 pending ops.
I20260924 19:58:49.764923 20980 pending_rounds.cc:77] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986591637950464 op_type: NO_OP noop_request { }
I20260924 19:58:49.765230 20980 raft_consensus.cc:2889] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:58:49.765538 20980 raft_consensus.cc:2272] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:49.770056 20980 ts_tablet_manager.cc:1916] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:49.771759 20438 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet a8d34fa77ecb4102919ff5fe6fd8412e
I20260924 19:58:49.774638 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet a8d34fa77ecb4102919ff5fe6fd8412e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:49 UTC) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:49.775100 20977 raft_consensus.cc:359] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.777580 20977 raft_consensus.cc:385] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.778007 20976 tablet_bootstrap.cc:492] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:49.778128 20977 raft_consensus.cc:740] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.779083 20977 consensus_queue.cc:260] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.784533 20700 tablet_service.cc:1558] Processing DeleteTablet for tablet a8d34fa77ecb4102919ff5fe6fd8412e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:49 UTC) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:49.785372 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet a8d34fa77ecb4102919ff5fe6fd8412e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:49 UTC) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:49.786032 20982 tablet_replica.cc:333] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:49.786657 20982 raft_consensus.cc:2243] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:49.787065 20982 raft_consensus.cc:2272] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:49.788686 20982 ts_tablet_manager.cc:1916] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:49.789903 20981 ts_tablet_manager.cc:1929] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:49.790251 20981 log.cc:1199] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/5fa9ceda714341e5b8eeb4144fea1342
I20260924 19:58:49.790825 20981 ts_tablet_manager.cc:1950] T 5fa9ceda714341e5b8eeb4144fea1342 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:49.792950 20976 tablet_bootstrap.cc:654] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:49.796469 20978 tablet_bootstrap.cc:654] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:49.792166 20980 ts_tablet_manager.cc:1929] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:49.800041 20980 log.cc:1199] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/a8d34fa77ecb4102919ff5fe6fd8412e
I20260924 19:58:49.800800 20980 ts_tablet_manager.cc:1950] T a8d34fa77ecb4102919ff5fe6fd8412e P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:49.804949 20976 tablet_bootstrap.cc:492] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
W20260924 19:58:49.805218 20271 catalog_manager.cc:5030] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet a8d34fa77ecb4102919ff5fe6fd8412e because the tablet was not found. No further retry: Not found: Tablet not found: a8d34fa77ecb4102919ff5fe6fd8412e
I20260924 19:58:49.805445 20976 ts_tablet_manager.cc:1403] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.008s
I20260924 19:58:49.808251 20976 raft_consensus.cc:359] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.808876 20976 raft_consensus.cc:385] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:49.809124 20976 raft_consensus.cc:740] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:49.809907 20976 consensus_queue.cc:260] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.816852 20976 ts_tablet_manager.cc:1434] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
W20260924 19:58:49.820866 20282 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_f25b80aa63f9428d849b3723d9f25a39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:49.826505 20982 ts_tablet_manager.cc:1929] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:49.826853 20982 log.cc:1199] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/a8d34fa77ecb4102919ff5fe6fd8412e
I20260924 19:58:49.827724 20977 ts_tablet_manager.cc:1434] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.130s user 0.006s sys 0.000s
I20260924 19:58:49.840692 20278 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40798 (request call id 23) took 4331 ms. Trace:
W20260924 19:58:49.840898 20277 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_6ac8d98b233146c495713e0cda0ac6b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:49.841063 20278 rpcz_store.cc:276] 0924 19:58:45.509760 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:45.510007 (+ 247us) service_pool.cc:224] Handling call
0924 19:58:45.555274 (+ 45267us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:49.840605 (+4285331us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1165226,"Hive Metastore.run_cpu_time_us":792,"Hive Metastore.run_wall_time_us":1204875}
I20260924 19:58:49.847920 20281 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40798 (request call id 18) took 4615 ms. Trace:
I20260924 19:58:49.848269 20281 rpcz_store.cc:276] 0924 19:58:45.232131 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:45.232267 (+ 136us) service_pool.cc:224] Handling call
0924 19:58:45.548533 (+316266us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:49.840021 (+4291488us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:49.846933 (+ 6912us) catalog_manager.cc:3764] Apply alter schema
0924 19:58:49.847849 (+ 916us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":6709,"Hive Metastore.run_cpu_time_us":2171,"Hive Metastore.run_wall_time_us":918162}
I20260924 19:58:49.854562 20281 master_service.cc:960] ReplaceTablet: received request to replace tablet 62f4d38a8da14252982bcac0e7df28c4 from {username='slave'} at 127.0.0.1:40798
W20260924 19:58:49.867270 20270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20260924 19:58:49.870381 20275 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7a523607282c4d1c85c7a42deabb470b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:49.874910 20284 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_2a7727ff525b44bc97232bb07f824105 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:49.878209 20978 tablet_bootstrap.cc:492] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:49.878623 20978 ts_tablet_manager.cc:1403] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.178s user 0.006s sys 0.004s
I20260924 19:58:49.882161 20981 tablet_replica.cc:333] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:49.882871 20981 raft_consensus.cc:2243] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:49.901414 20281 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 62f4d38a8da14252982bcac0e7df28c4
I20260924 19:58:49.903038 20281 catalog_manager.cc:6934] ReplaceTablet: tablet 62f4d38a8da14252982bcac0e7df28c4 deleted and replaced by tablet 8438dd32ff05426e947d9033981a3ea4
I20260924 19:58:49.905640 20977 tablet_bootstrap.cc:492] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:49.904276 20279 catalog_manager.cc:5838] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0055218dbcbd48b58018e9774e59e14c (127.17.2.131). New cstate: current_term: 1 leader_uuid: "0055218dbcbd48b58018e9774e59e14c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:49.881618 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 5fa9ceda714341e5b8eeb4144fea1342 (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:49.915740 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet 62f4d38a8da14252982bcac0e7df28c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8438dd32ff05426e947d9033981a3ea4) from {username='slave'} at 127.0.0.1:48342
W20260924 19:58:49.917418 20271 catalog_manager.cc:5043] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet 62f4d38a8da14252982bcac0e7df28c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f4d38a8da14252982bcac0e7df28c4 already in progress: creating tablet
I20260924 19:58:49.920455 20555 tablet_service.cc:1511] Processing CreateTablet for tablet 82554a94caa44720b56d1ee51dd11163 (DEFAULT_TABLE table=default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.921452 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82554a94caa44720b56d1ee51dd11163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.943328 20283 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40798 (request call id 19) took 4571 ms. Trace:
I20260924 19:58:49.943686 20283 rpcz_store.cc:276] 0924 19:58:45.371546 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:45.371789 (+ 243us) service_pool.cc:224] Handling call
0924 19:58:45.377932 (+ 6143us) spinlock_profiling.cc:243] Waited 4.66 ms on lock 0x7b54000104d4. stack: 00007fe7ffab1323 00007fe7ffab1220 00007fe7ff655574 0000000000516964 00000000005168f9 00007fe806baf5c0 00007fe806c4e6fa 00007fe8089be98c 00007fe808abe7e6 00007fe806673634 00007fe8066735c2 00007fe8066734f1 00007fe80667346c 00007fe806672722 00007fe80491409c 00007fe8049134d6
0924 19:58:45.550135 (+172203us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:49.840342 (+4290207us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:49.942299 (+101957us) catalog_manager.cc:3764] Apply alter schema
0924 19:58:49.943260 (+ 961us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":923539,"Hive Metastore.run_cpu_time_us":1873,"Hive Metastore.run_wall_time_us":246591,"mutex_wait_us":69282,"spinlock_wait_cycles":10726784}
I20260924 19:58:49.887563 21105 raft_consensus.cc:493] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:49.952313 20982 ts_tablet_manager.cc:1950] T a8d34fa77ecb4102919ff5fe6fd8412e P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:49.951992 21105 raft_consensus.cc:515] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.954705 21105 leader_election.cc:290] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:49.904091 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet 62f4d38a8da14252982bcac0e7df28c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8438dd32ff05426e947d9033981a3ea4) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:49.933624 20847 tablet_service.cc:1511] Processing CreateTablet for tablet 82554a94caa44720b56d1ee51dd11163 (DEFAULT_TABLE table=default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:49.957522 20847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82554a94caa44720b56d1ee51dd11163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:49.958204 20982 tablet_replica.cc:333] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:49.958894 20982 raft_consensus.cc:2243] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:49.959329 20982 raft_consensus.cc:2272] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:49.960476 20553 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fa9ceda714341e5b8eeb4144fea1342 with delete_type TABLET_DATA_DELETED (replaced by tablet 82554a94caa44720b56d1ee51dd11163) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:49.963037 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5fa9ceda714341e5b8eeb4144fea1342" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:49.963183 20977 tablet_bootstrap.cc:654] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:49.963706 20982 ts_tablet_manager.cc:1916] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:49.963661 20574 raft_consensus.cc:2468] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
W20260924 19:58:49.967118 20804 leader_election.cc:343] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 5fa9ceda714341e5b8eeb4144fea1342
I20260924 19:58:49.967921 20280 master_service.cc:960] ReplaceTablet: received request to replace tablet 5fa9ceda714341e5b8eeb4144fea1342 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:49.970810 20804 leader_election.cc:304] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters: af20c71727a14e398a217d4623d1399a
I20260924 19:58:49.971868 21105 raft_consensus.cc:2804] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:49.972180 21105 raft_consensus.cc:493] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:49.972469 21105 raft_consensus.cc:3060] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:49.977984 20981 raft_consensus.cc:2272] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:49.979878 20981 ts_tablet_manager.cc:1916] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:49.980489 20980 tablet_replica.cc:333] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:49.983879 20980 raft_consensus.cc:2243] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:49.985095 20980 raft_consensus.cc:2272] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:49.986385 21105 raft_consensus.cc:515] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.989075 20980 ts_tablet_manager.cc:1916] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:49.997401 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5fa9ceda714341e5b8eeb4144fea1342" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
W20260924 19:58:49.998935 20804 leader_election.cc:343] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:49.960711 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet a8d34fa77ecb4102919ff5fe6fd8412e (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:49.893167 20978 raft_consensus.cc:359] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.914757 20700 tablet_service.cc:1558] Processing DeleteTablet for tablet 62f4d38a8da14252982bcac0e7df28c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8438dd32ff05426e947d9033981a3ea4) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:50.003181 20982 ts_tablet_manager.cc:1929] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.003676 20982 log.cc:1199] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/62f4d38a8da14252982bcac0e7df28c4
I20260924 19:58:50.003757 20978 raft_consensus.cc:385] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.004194 20978 raft_consensus.cc:740] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.004456 20982 ts_tablet_manager.cc:1950] T 62f4d38a8da14252982bcac0e7df28c4 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:49.949736 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fa9ceda714341e5b8eeb4144fea1342 with delete_type TABLET_DATA_DELETED (replaced by tablet 82554a94caa44720b56d1ee51dd11163) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:50.005146 20978 consensus_queue.cc:260] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:49.931665 20701 tablet_service.cc:1511] Processing CreateTablet for tablet 82554a94caa44720b56d1ee51dd11163 (DEFAULT_TABLE table=default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.007385 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82554a94caa44720b56d1ee51dd11163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.012214 20978 ts_tablet_manager.cc:1434] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.133s user 0.007s sys 0.000s
I20260924 19:58:50.013415 21105 leader_election.cc:290] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:50.015981 20977 tablet_bootstrap.cc:492] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:50.016402 20977 ts_tablet_manager.cc:1403] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.111s user 0.009s sys 0.000s
I20260924 19:58:50.018703 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 62f4d38a8da14252982bcac0e7df28c4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:50.019145 20977 raft_consensus.cc:359] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.019796 20977 raft_consensus.cc:385] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.020083 20977 raft_consensus.cc:740] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.022776 20977 consensus_queue.cc:260] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:58:50.023495 20804 leader_election.cc:343] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 5fa9ceda714341e5b8eeb4144fea1342
I20260924 19:58:50.023942 20804 leader_election.cc:304] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:58:50.036357 20977 ts_tablet_manager.cc:1434] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260924 19:58:50.046012 20278 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_92546dd5153b4d58af6fc19d4dee39f0" } modify_external_catalogs: true
I20260924 19:58:50.046859 20977 tablet_bootstrap.cc:492] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:50.046859 20278 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_92546dd5153b4d58af6fc19d4dee39f0" } modify_external_catalogs: true
I20260924 19:58:50.048449 20978 tablet_bootstrap.cc:492] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:50.057350 20977 tablet_bootstrap.cc:654] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.057498 20281 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_1ed870b0a11e446c86e6a4c6a15e81b6" } new_table_name: "default.Table_c8b8f1cd14774827a2cc045e0ffbf35d" modify_external_catalogs: true
I20260924 19:58:50.064049 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet 62f4d38a8da14252982bcac0e7df28c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8438dd32ff05426e947d9033981a3ea4) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:50.064654 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fa9ceda714341e5b8eeb4144fea1342 with delete_type TABLET_DATA_DELETED (replaced by tablet 82554a94caa44720b56d1ee51dd11163) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:50.065337 20982 tablet_replica.cc:333] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:50.065642 21105 raft_consensus.cc:2749] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 19:58:50.066721 20982 raft_consensus.cc:2243] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:50.067257 20982 raft_consensus.cc:2272] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.069275 20982 ts_tablet_manager.cc:1916] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.070624 20977 tablet_bootstrap.cc:492] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:50.071635 20977 ts_tablet_manager.cc:1403] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.000s
I20260924 19:58:50.073065 20976 tablet_bootstrap.cc:492] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:50.075503 20977 raft_consensus.cc:359] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.078440 20977 raft_consensus.cc:385] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.079078 20977 raft_consensus.cc:740] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
W20260924 19:58:50.079506 20270 catalog_manager.cc:5030] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): delete failed for tablet 62f4d38a8da14252982bcac0e7df28c4 because the tablet was not found. No further retry: Not found: Tablet not found: 62f4d38a8da14252982bcac0e7df28c4
I20260924 19:58:50.079087 20976 tablet_bootstrap.cc:654] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.080552 20977 consensus_queue.cc:260] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.086690 20978 tablet_bootstrap.cc:654] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.089802 20977 ts_tablet_manager.cc:1434] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260924 19:58:50.092672 20977 tablet_bootstrap.cc:492] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:50.102985 20977 tablet_bootstrap.cc:654] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.109171 20980 ts_tablet_manager.cc:1929] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.109550 20980 log.cc:1199] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/5fa9ceda714341e5b8eeb4144fea1342
I20260924 19:58:50.110170 20980 ts_tablet_manager.cc:1950] T 5fa9ceda714341e5b8eeb4144fea1342 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:50.119871 20977 tablet_bootstrap.cc:492] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:50.111120 20982 ts_tablet_manager.cc:1929] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.120814 20982 log.cc:1199] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/5fa9ceda714341e5b8eeb4144fea1342
I20260924 19:58:50.121567 20982 ts_tablet_manager.cc:1950] T 5fa9ceda714341e5b8eeb4144fea1342 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:50.121668 20977 ts_tablet_manager.cc:1403] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.002s
I20260924 19:58:50.124990 20977 raft_consensus.cc:359] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.127795 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 5fa9ceda714341e5b8eeb4144fea1342 (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:50.129316 20977 raft_consensus.cc:385] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.130493 20977 raft_consensus.cc:740] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.131732 20977 consensus_queue.cc:260] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.135644 20977 ts_tablet_manager.cc:1434] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20260924 19:58:50.137845 20977 tablet_bootstrap.cc:492] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:50.139096 20283 master_service.cc:960] ReplaceTablet: received request to replace tablet 8ebcdc9a748549f7bddb69bb5b99b315 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:50.148494 20279 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_14cbcbeb7f98417197bb13b1f8b5af49" } new_table_name: "default.Table_77566e1a46ec4a4ba7dde05d53b7c5cc" modify_external_catalogs: true
I20260924 19:58:50.151638 20981 ts_tablet_manager.cc:1929] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.152036 20981 log.cc:1199] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/a8d34fa77ecb4102919ff5fe6fd8412e
I20260924 19:58:50.152671 20981 ts_tablet_manager.cc:1950] T a8d34fa77ecb4102919ff5fe6fd8412e P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:50.154341 20977 tablet_bootstrap.cc:654] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.154819 20978 tablet_bootstrap.cc:492] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:50.155243 20978 ts_tablet_manager.cc:1403] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.107s user 0.010s sys 0.000s
I20260924 19:58:50.157651 20978 raft_consensus.cc:359] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.158263 20978 raft_consensus.cc:385] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.158502 20978 raft_consensus.cc:740] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.173543 20977 tablet_bootstrap.cc:492] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:50.173949 20977 ts_tablet_manager.cc:1403] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.002s
I20260924 19:58:50.179381 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 5fa9ceda714341e5b8eeb4144fea1342 (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:50.159123 20978 consensus_queue.cc:260] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.178815 20977 raft_consensus.cc:359] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.182271 20977 raft_consensus.cc:385] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.182595 20977 raft_consensus.cc:740] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.183465 20978 ts_tablet_manager.cc:1434] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260924 19:58:50.184545 20977 consensus_queue.cc:260] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.190402 20977 ts_tablet_manager.cc:1434] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260924 19:58:50.192819 20283 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 8ebcdc9a748549f7bddb69bb5b99b315
I20260924 19:58:50.194944 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet 8ebcdc9a748549f7bddb69bb5b99b315 with delete_type TABLET_DATA_DELETED (replaced by tablet e2719ada60a143c6862e90d81f9c2d82) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:50.169575 20981 tablet_replica.cc:333] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:50.197842 20283 catalog_manager.cc:6934] ReplaceTablet: tablet 8ebcdc9a748549f7bddb69bb5b99b315 deleted and replaced by tablet e2719ada60a143c6862e90d81f9c2d82
I20260924 19:58:50.197928 20847 tablet_service.cc:1558] Processing DeleteTablet for tablet 8ebcdc9a748549f7bddb69bb5b99b315 with delete_type TABLET_DATA_DELETED (replaced by tablet e2719ada60a143c6862e90d81f9c2d82) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:50.198660 20982 tablet_replica.cc:333] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:50.199332 20982 raft_consensus.cc:2243] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:50.199774 20982 raft_consensus.cc:2272] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.199782 20283 master_service.cc:960] ReplaceTablet: received request to replace tablet 82554a94caa44720b56d1ee51dd11163 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:50.201495 20982 ts_tablet_manager.cc:1916] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.207386 20980 tablet_replica.cc:333] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:50.208156 20980 raft_consensus.cc:2243] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:50.208642 20980 raft_consensus.cc:2272] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.210781 20980 ts_tablet_manager.cc:1916] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.212265 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet 62f4d38a8da14252982bcac0e7df28c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8438dd32ff05426e947d9033981a3ea4) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:50.215727 20976 tablet_bootstrap.cc:492] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:50.216223 20976 ts_tablet_manager.cc:1403] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.143s user 0.027s sys 0.017s
I20260924 19:58:50.219313 20976 raft_consensus.cc:359] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.220081 20976 raft_consensus.cc:385] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.220099 20981 raft_consensus.cc:2243] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:50.220417 20976 raft_consensus.cc:740] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.220641 20981 raft_consensus.cc:2272] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.222731 20981 ts_tablet_manager.cc:1916] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.224673 20976 consensus_queue.cc:260] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.239858 20976 ts_tablet_manager.cc:1434] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260924 19:58:50.241268 20980 ts_tablet_manager.cc:1929] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.241695 20980 log.cc:1199] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/8ebcdc9a748549f7bddb69bb5b99b315
I20260924 19:58:50.242442 20980 ts_tablet_manager.cc:1950] T 8ebcdc9a748549f7bddb69bb5b99b315 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:50.244921 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 8ebcdc9a748549f7bddb69bb5b99b315 (table default.table_14cbcbeb7f98417197bb13b1f8b5af49 [id=c2d09462b2b14d1a862ef2b18152c4e1]) successfully deleted
I20260924 19:58:50.245671 20283 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 82554a94caa44720b56d1ee51dd11163
I20260924 19:58:50.248018 20555 tablet_service.cc:1511] Processing CreateTablet for tablet 8438dd32ff05426e947d9033981a3ea4 (DEFAULT_TABLE table=default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.248961 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8438dd32ff05426e947d9033981a3ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.249504 20980 tablet_replica.cc:333] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:50.250174 20980 raft_consensus.cc:2243] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:50.251896 20980 raft_consensus.cc:2272] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.254621 20980 ts_tablet_manager.cc:1916] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.268770 20553 tablet_service.cc:1558] Processing DeleteTablet for tablet 82554a94caa44720b56d1ee51dd11163 with delete_type TABLET_DATA_DELETED (replaced by tablet 65e0685bc4564703b88a5247511686ec) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:50.270327 20982 ts_tablet_manager.cc:1929] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.270692 20982 log.cc:1199] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/8ebcdc9a748549f7bddb69bb5b99b315
I20260924 19:58:50.279987 20980 ts_tablet_manager.cc:1929] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.280359 20980 log.cc:1199] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/62f4d38a8da14252982bcac0e7df28c4
I20260924 19:58:50.281577 20978 tablet_bootstrap.cc:492] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:50.284590 20847 tablet_service.cc:1558] Processing DeleteTablet for tablet 82554a94caa44720b56d1ee51dd11163 with delete_type TABLET_DATA_DELETED (replaced by tablet 65e0685bc4564703b88a5247511686ec) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:50.284610 20701 tablet_service.cc:1558] Processing DeleteTablet for tablet 8ebcdc9a748549f7bddb69bb5b99b315 with delete_type TABLET_DATA_DELETED (replaced by tablet e2719ada60a143c6862e90d81f9c2d82) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:50.285109 20980 ts_tablet_manager.cc:1950] T 62f4d38a8da14252982bcac0e7df28c4 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:50.285655 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet a8d34fa77ecb4102919ff5fe6fd8412e (table default.table_d74dff5eb61a4b6fb10a13c1ec755df9 [id=fb6b0484c7d2481eb450f18797dac6b1]) successfully deleted
I20260924 19:58:50.287343 20980 tablet_replica.cc:333] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:50.287521 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 82554a94caa44720b56d1ee51dd11163 with delete_type TABLET_DATA_DELETED (replaced by tablet 65e0685bc4564703b88a5247511686ec) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:50.287744 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 62f4d38a8da14252982bcac0e7df28c4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:50.288005 20980 raft_consensus.cc:2243] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:50.288396 20980 raft_consensus.cc:2272] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.288748 20701 tablet_service.cc:1511] Processing CreateTablet for tablet 8438dd32ff05426e947d9033981a3ea4 (DEFAULT_TABLE table=default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.290642 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8438dd32ff05426e947d9033981a3ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.291501 20847 tablet_service.cc:1511] Processing CreateTablet for tablet 8438dd32ff05426e947d9033981a3ea4 (DEFAULT_TABLE table=default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.292609 20847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8438dd32ff05426e947d9033981a3ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.294548 20980 ts_tablet_manager.cc:1916] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.308885 20977 tablet_bootstrap.cc:492] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:50.310986 20982 ts_tablet_manager.cc:1950] T 8ebcdc9a748549f7bddb69bb5b99b315 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:50.313391 20982 tablet_replica.cc:333] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:50.313668 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 8ebcdc9a748549f7bddb69bb5b99b315 (table default.table_14cbcbeb7f98417197bb13b1f8b5af49 [id=c2d09462b2b14d1a862ef2b18152c4e1]) successfully deleted
I20260924 19:58:50.314239 20977 tablet_bootstrap.cc:654] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.282522 20283 catalog_manager.cc:6934] ReplaceTablet: tablet 82554a94caa44720b56d1ee51dd11163 deleted and replaced by tablet 65e0685bc4564703b88a5247511686ec
I20260924 19:58:50.320639 20978 tablet_bootstrap.cc:654] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.333204 20980 ts_tablet_manager.cc:1929] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.333575 20980 log.cc:1199] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/82554a94caa44720b56d1ee51dd11163
I20260924 19:58:50.334189 20980 ts_tablet_manager.cc:1950] T 82554a94caa44720b56d1ee51dd11163 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:50.336799 20977 tablet_bootstrap.cc:492] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:50.337200 20977 ts_tablet_manager.cc:1403] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.004s
I20260924 19:58:50.314042 20982 raft_consensus.cc:2243] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:50.338814 20982 raft_consensus.cc:2272] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.339520 20977 raft_consensus.cc:359] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.340108 20977 raft_consensus.cc:385] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.340329 20977 raft_consensus.cc:740] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.342127 20977 consensus_queue.cc:260] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.349129 20982 ts_tablet_manager.cc:1916] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.352599 20977 ts_tablet_manager.cc:1434] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260924 19:58:50.361012 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 82554a94caa44720b56d1ee51dd11163 (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:50.372952 20978 tablet_bootstrap.cc:492] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:50.373402 20978 ts_tablet_manager.cc:1403] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.093s user 0.011s sys 0.000s
I20260924 19:58:50.384207 20978 raft_consensus.cc:359] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.385011 20978 raft_consensus.cc:385] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.385289 20978 raft_consensus.cc:740] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.386097 20978 consensus_queue.cc:260] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.411998 20981 ts_tablet_manager.cc:1929] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.412421 20981 log.cc:1199] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/62f4d38a8da14252982bcac0e7df28c4
I20260924 19:58:50.413262 20981 ts_tablet_manager.cc:1950] T 62f4d38a8da14252982bcac0e7df28c4 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:50.416563 20978 ts_tablet_manager.cc:1434] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.043s user 0.007s sys 0.000s
I20260924 19:58:50.423743 20978 tablet_bootstrap.cc:492] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:50.445425 20280 catalog_manager.cc:5938] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667) does not have the latest schema for tablet 8438dd32ff05426e947d9033981a3ea4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]). Expected version 1 got 0
I20260924 19:58:50.448170 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 62f4d38a8da14252982bcac0e7df28c4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
W20260924 19:58:50.449115 20271 catalog_manager.cc:5030] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet 5fa9ceda714341e5b8eeb4144fea1342 because the tablet was not found. No further retry: Not found: Tablet not found: 5fa9ceda714341e5b8eeb4144fea1342
I20260924 19:58:50.449379 20981 tablet_replica.cc:333] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:50.450057 20981 raft_consensus.cc:2243] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:50.450496 20981 raft_consensus.cc:2272] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.453054 20981 ts_tablet_manager.cc:1916] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.456972 20978 tablet_bootstrap.cc:654] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
W20260924 19:58:50.459903 20280 catalog_manager.cc:6038] Failed to send AlterTable RPC for tablet 8438dd32ff05426e947d9033981a3ea4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 8438dd32ff05426e947d9033981a3ea4 consensus state has no leader
W20260924 19:58:50.460527 20270 consensus_peers.cc:597] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 -> Peer 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311): Couldn't send request to peer 022c1898fcbe4856aa680c26047f6934. Status: Network error: Client connection negotiation failed: client connection to 127.17.2.188:35311: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260924 19:58:50.462026 20982 ts_tablet_manager.cc:1929] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.462380 20982 log.cc:1199] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/82554a94caa44720b56d1ee51dd11163
I20260924 19:58:50.462988 20982 ts_tablet_manager.cc:1950] T 82554a94caa44720b56d1ee51dd11163 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:50.463892 20976 tablet_bootstrap.cc:492] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
W20260924 19:58:50.483690 20269 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 8438dd32ff05426e947d9033981a3ea4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8438dd32ff05426e947d9033981a3ea4 consensus state has no leader
I20260924 19:58:50.484692 20976 tablet_bootstrap.cc:654] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.489276 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 82554a94caa44720b56d1ee51dd11163 (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:50.498672 20282 catalog_manager.cc:2213] created HMS catalog entry for table [id=a41c0a152a10436fb22b0c1b14abd300]
W20260924 19:58:48.444981 21032 debug-util.cc:398] Leaking SignalData structure 0x7b08000292a0 after lost signal to thread 20959
W20260924 19:58:50.510603 21032 kernel_stack_watchdog.cc:198] Thread 20959 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 443ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260924 19:58:46.997997 21039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:58:50.512014 20959 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 3.513s user 0.411s sys 0.808s
W20260924 19:58:50.512428 20959 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 3.513s user 0.411s sys 0.808s
I20260924 19:58:50.512763 20976 tablet_bootstrap.cc:492] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:50.513020 20959 server_base.cc:1061] running on GCE node
I20260924 19:58:50.513195 20976 ts_tablet_manager.cc:1403] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.006s
W20260924 19:58:50.514328 21046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:58:50.515656 21105 consensus_queue.cc:1048] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:50.516750 20976 raft_consensus.cc:359] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.517343 20976 raft_consensus.cc:385] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.517576 20976 raft_consensus.cc:740] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.518208 20976 consensus_queue.cc:260] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.520288 20976 ts_tablet_manager.cc:1434] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260924 19:58:50.524605 20959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:58:50.542765 20282 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 24) took 4973 ms. Trace:
I20260924 19:58:50.553500 20978 tablet_bootstrap.cc:492] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:50.554255 20978 ts_tablet_manager.cc:1403] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.131s user 0.013s sys 0.000s
I20260924 19:58:50.557483 20978 raft_consensus.cc:359] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.558101 20978 raft_consensus.cc:385] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.558360 20978 raft_consensus.cc:740] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.566802 20978 consensus_queue.cc:260] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.569584 20978 ts_tablet_manager.cc:1434] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260924 19:58:50.572224 20981 ts_tablet_manager.cc:1929] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.572566 20981 log.cc:1199] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/8ebcdc9a748549f7bddb69bb5b99b315
I20260924 19:58:50.573128 20981 ts_tablet_manager.cc:1950] T 8ebcdc9a748549f7bddb69bb5b99b315 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:50.585809 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 8ebcdc9a748549f7bddb69bb5b99b315 (table default.table_14cbcbeb7f98417197bb13b1f8b5af49 [id=c2d09462b2b14d1a862ef2b18152c4e1]) successfully deleted
I20260924 19:58:50.586021 20981 tablet_replica.cc:333] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:50.586655 20981 raft_consensus.cc:2243] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:50.587051 20981 raft_consensus.cc:2272] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.543130 20282 rpcz_store.cc:276] 0924 19:58:45.569229 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:45.569364 (+ 135us) service_pool.cc:224] Handling call
0924 19:58:49.835796 (+4266432us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:49.842636 (+ 6840us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:50.498627 (+655991us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:50.542103 (+ 43476us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:58:50.542406 (+ 303us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:58:50.542701 (+ 295us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:50.500487 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:50.500876 (+ 389us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:50.500898 (+ 22us) write_op.cc:435] Acquired schema lock
0924 19:58:50.500912 (+ 14us) tablet.cc:662] Decoding operations
0924 19:58:50.501154 (+ 242us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:50.501192 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:50.501214 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:50.501395 (+ 181us) tablet.cc:701] Row locks acquired
0924 19:58:50.501407 (+ 12us) write_op.cc:260] PREPARE: finished
0924 19:58:50.501517 (+ 110us) write_op.cc:270] Start()
0924 19:58:50.501603 (+ 86us) write_op.cc:276] Timestamp: P: 1790279930501462 usec, L: 0
0924 19:58:50.501617 (+ 14us) op_driver.cc:348] REPLICATION: starting
0924 19:58:50.502055 (+ 438us) log.cc:844] Serialized 432 byte log entry
0924 19:58:50.528794 (+ 26739us) op_driver.cc:464] REPLICATION: finished
0924 19:58:50.529235 (+ 441us) write_op.cc:301] APPLY: starting
0924 19:58:50.529290 (+ 55us) tablet.cc:1394] starting BulkCheckPresence
0924 19:58:50.529389 (+ 99us) tablet.cc:1397] finished BulkCheckPresence
0924 19:58:50.529496 (+ 107us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:58:50.529885 (+ 389us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:58:50.530010 (+ 125us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:58:50.530045 (+ 35us) write_op.cc:312] APPLY: finished
0924 19:58:50.530226 (+ 181us) log.cc:844] Serialized 28 byte log entry
0924 19:58:50.530367 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:50.530432 (+ 65us) write_op.cc:454] Released schema lock
0924 19:58:50.530462 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:58:50.530687 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:50.530710 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":6533,"Hive Metastore.run_cpu_time_us":610,"Hive Metastore.run_wall_time_us":648127,"child_traces":[["op",{"apply.queue_time_us":292,"apply.run_cpu_time_us":1424,"apply.run_wall_time_us":1423,"num_ops":2,"prepare.queue_time_us":162,"prepare.run_cpu_time_us":1854,"prepare.run_wall_time_us":1855,"raft.queue_time_us":2677,"raft.run_cpu_time_us":685,"raft.run_wall_time_us":684,"replication_time_us":27075}]]}
I20260924 19:58:50.583683 20847 tablet_service.cc:1558] Processing DeleteTablet for tablet 82554a94caa44720b56d1ee51dd11163 with delete_type TABLET_DATA_DELETED (replaced by tablet 65e0685bc4564703b88a5247511686ec) from {username='slave'} at 127.0.0.1:60248
W20260924 19:58:50.591432 20270 catalog_manager.cc:5030] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): delete failed for tablet 82554a94caa44720b56d1ee51dd11163 because the tablet was not found. No further retry: Not found: Tablet not found: 82554a94caa44720b56d1ee51dd11163
I20260924 19:58:50.595536 20282 master_service.cc:960] ReplaceTablet: received request to replace tablet 4b925a15a6074be0ae5d2c43a7ab83d1 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:50.599390 20555 tablet_service.cc:1511] Processing CreateTablet for tablet 65e0685bc4564703b88a5247511686ec (DEFAULT_TABLE table=default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.600428 20555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e0685bc4564703b88a5247511686ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.602669 20574 raft_consensus.cc:3060] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:50.604596 20981 ts_tablet_manager.cc:1916] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.611980 21105 consensus_queue.cc:1048] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:50.615332 20701 tablet_service.cc:1511] Processing CreateTablet for tablet 65e0685bc4564703b88a5247511686ec (DEFAULT_TABLE table=default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.616639 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e0685bc4564703b88a5247511686ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.627790 20283 catalog_manager.cc:5938] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667) does not have the latest schema for tablet 8438dd32ff05426e947d9033981a3ea4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]). Expected version 1 got 0
I20260924 19:58:50.631673 20981 ts_tablet_manager.cc:1929] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:50.632160 20981 log.cc:1199] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/82554a94caa44720b56d1ee51dd11163
I20260924 19:58:50.632766 20981 ts_tablet_manager.cc:1950] T 82554a94caa44720b56d1ee51dd11163 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:50.635426 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 82554a94caa44720b56d1ee51dd11163 (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]) successfully deleted
I20260924 19:58:50.635933 20280 catalog_manager.cc:5938] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601) does not have the latest schema for tablet 8438dd32ff05426e947d9033981a3ea4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]). Expected version 1 got 0
I20260924 19:58:50.638207 20280 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c40348984314595b6e6e49c4f03f35e from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:50.654515 20846 tablet_service.cc:1511] Processing CreateTablet for tablet e2719ada60a143c6862e90d81f9c2d82 (DEFAULT_TABLE table=default.table_14cbcbeb7f98417197bb13b1f8b5af49 [id=c2d09462b2b14d1a862ef2b18152c4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.655603 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2719ada60a143c6862e90d81f9c2d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.658314 20553 tablet_service.cc:1511] Processing CreateTablet for tablet e2719ada60a143c6862e90d81f9c2d82 (DEFAULT_TABLE table=default.table_14cbcbeb7f98417197bb13b1f8b5af49 [id=c2d09462b2b14d1a862ef2b18152c4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.659325 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2719ada60a143c6862e90d81f9c2d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.661846 20847 tablet_service.cc:1511] Processing CreateTablet for tablet 65e0685bc4564703b88a5247511686ec (DEFAULT_TABLE table=default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.662781 20978 tablet_bootstrap.cc:492] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:50.663072 20702 tablet_service.cc:1511] Processing CreateTablet for tablet e2719ada60a143c6862e90d81f9c2d82 (DEFAULT_TABLE table=default.table_14cbcbeb7f98417197bb13b1f8b5af49 [id=c2d09462b2b14d1a862ef2b18152c4e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.663420 20959 hybrid_clock.cc:648] HybridClock initialized: now 1790279930663369 us; error 44 us; skew 500 ppm
I20260924 19:58:50.664039 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2719ada60a143c6862e90d81f9c2d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.664523 20959 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:58:50.676261 20847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e0685bc4564703b88a5247511686ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.684441 20978 tablet_bootstrap.cc:654] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.725150 20554 tablet_service.cc:1511] Processing CreateTablet for tablet e01e0614ba6e4404a393ed67b6499c4b (DEFAULT_TABLE table=default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.725309 20701 tablet_service.cc:1511] Processing CreateTablet for tablet e01e0614ba6e4404a393ed67b6499c4b (DEFAULT_TABLE table=default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.726341 20554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01e0614ba6e4404a393ed67b6499c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.726306 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01e0614ba6e4404a393ed67b6499c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.728870 20848 tablet_service.cc:1511] Processing CreateTablet for tablet e01e0614ba6e4404a393ed67b6499c4b (DEFAULT_TABLE table=default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:50.729952 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e01e0614ba6e4404a393ed67b6499c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:50.756413 20977 tablet_bootstrap.cc:492] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:50.762420 20977 tablet_bootstrap.cc:654] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.763674 20959 webserver.cc:533] Webserver started at http://127.17.2.188:36753/ using document root <none> and password file <none>
I20260924 19:58:50.765151 20959 fs_manager.cc:362] Metadata directory not provided
I20260924 19:58:50.765470 20959 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:58:50.769493 20277 catalog_manager.cc:2213] created HMS catalog entry for table [id=46a1524a039b484d9118dee4b62cb8f9]
I20260924 19:58:50.777321 20555 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 with delete_type TABLET_DATA_DELETED (replaced by tablet b8ec997ee07b4953847c944ee1e9ec64) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:50.778173 20980 tablet_replica.cc:333] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:50.778864 20980 raft_consensus.cc:2243] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:50.779724 20980 pending_rounds.cc:70] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: Trying to abort 1 pending ops.
I20260924 19:58:50.779963 20980 pending_rounds.cc:77] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986592022847488 op_type: NO_OP noop_request { }
I20260924 19:58:50.780290 20980 raft_consensus.cc:2889] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:58:50.780664 20980 raft_consensus.cc:2272] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.783103 20980 ts_tablet_manager.cc:1916] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.787345 20977 tablet_bootstrap.cc:492] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:50.787781 20977 ts_tablet_manager.cc:1403] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.001s
I20260924 19:58:50.789467 20976 tablet_bootstrap.cc:492] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:50.795068 20976 tablet_bootstrap.cc:654] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.801721 20277 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 26) took 5089 ms. Trace:
I20260924 19:58:50.802248 20277 rpcz_store.cc:276] 0924 19:58:45.712224 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:45.712904 (+ 680us) service_pool.cc:224] Handling call
0924 19:58:49.841354 (+4128450us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:49.860791 (+ 19437us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:50.769441 (+908650us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:50.800787 (+ 31346us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:58:50.801256 (+ 469us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:58:50.801638 (+ 382us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:50.781412 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:50.781931 (+ 519us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:50.781953 (+ 22us) write_op.cc:435] Acquired schema lock
0924 19:58:50.781967 (+ 14us) tablet.cc:662] Decoding operations
0924 19:58:50.782252 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:50.782290 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:50.782312 (+ 22us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:50.782509 (+ 197us) tablet.cc:701] Row locks acquired
0924 19:58:50.782631 (+ 122us) write_op.cc:260] PREPARE: finished
0924 19:58:50.782762 (+ 131us) write_op.cc:270] Start()
0924 19:58:50.782854 (+ 92us) write_op.cc:276] Timestamp: P: 1790279930782713 usec, L: 0
0924 19:58:50.782870 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:58:50.783539 (+ 669us) log.cc:844] Serialized 432 byte log entry
0924 19:58:50.797974 (+ 14435us) op_driver.cc:464] REPLICATION: finished
0924 19:58:50.799088 (+ 1114us) write_op.cc:301] APPLY: starting
0924 19:58:50.799149 (+ 61us) tablet.cc:1394] starting BulkCheckPresence
0924 19:58:50.799292 (+ 143us) tablet.cc:1397] finished BulkCheckPresence
0924 19:58:50.799388 (+ 96us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:58:50.799777 (+ 389us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:58:50.799917 (+ 140us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:58:50.799980 (+ 63us) write_op.cc:312] APPLY: finished
0924 19:58:50.800218 (+ 238us) log.cc:844] Serialized 28 byte log entry
0924 19:58:50.800399 (+ 181us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:50.800491 (+ 92us) write_op.cc:454] Released schema lock
0924 19:58:50.800532 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":636804,"Hive Metastore.run_cpu_time_us":564,"Hive Metastore.run_wall_time_us":270887,"child_traces":[["op",{"apply.queue_time_us":958,"num_ops":2,"prepare.queue_time_us":9820,"raft.queue_time_us":144,"raft.run_cpu_time_us":904,"raft.run_wall_time_us":905,"replication_time_us":14991}]]}
I20260924 19:58:50.811578 20977 raft_consensus.cc:359] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.812423 20977 raft_consensus.cc:385] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.812706 20977 raft_consensus.cc:740] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.817060 20976 tablet_bootstrap.cc:492] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:50.817489 20976 ts_tablet_manager.cc:1403] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.001s
I20260924 19:58:50.822576 20847 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 with delete_type TABLET_DATA_DELETED (replaced by tablet b8ec997ee07b4953847c944ee1e9ec64) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:50.822440 20977 consensus_queue.cc:260] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.823405 20982 tablet_replica.cc:333] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:50.824069 20982 raft_consensus.cc:2243] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:50.826195 20959 fs_manager.cc:714] Time spent opening directory manager: real 0.056s user 0.006s sys 0.011s
I20260924 19:58:50.830214 20976 raft_consensus.cc:359] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.832670 20976 raft_consensus.cc:385] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.833832 20976 raft_consensus.cc:740] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.836923 20700 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 with delete_type TABLET_DATA_DELETED (replaced by tablet b8ec997ee07b4953847c944ee1e9ec64) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:50.836063 20976 consensus_queue.cc:260] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.837653 20981 tablet_replica.cc:333] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:50.838316 20981 raft_consensus.cc:2243] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:50.838747 20981 pending_rounds.cc:70] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: Trying to abort 1 pending ops.
I20260924 19:58:50.838950 20981 pending_rounds.cc:77] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986592022847488 op_type: NO_OP noop_request { }
I20260924 19:58:50.839358 20981 raft_consensus.cc:2889] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:58:50.839690 20981 raft_consensus.cc:2272] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.841871 20977 ts_tablet_manager.cc:1434] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.054s user 0.008s sys 0.000s
I20260924 19:58:50.842651 20981 ts_tablet_manager.cc:1916] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.846908 21130 raft_consensus.cc:493] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:50.847371 21130 raft_consensus.cc:515] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.849192 21130 leader_election.cc:290] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:50.853116 20277 catalog_manager.cc:5938] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667) does not have the latest schema for tablet 65e0685bc4564703b88a5247511686ec (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]). Expected version 1 got 0
I20260924 19:58:50.853718 20976 ts_tablet_manager.cc:1434] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.036s user 0.006s sys 0.001s
I20260924 19:58:50.866169 20977 tablet_bootstrap.cc:492] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:50.876456 20978 tablet_bootstrap.cc:492] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:50.876866 20978 ts_tablet_manager.cc:1403] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.214s user 0.014s sys 0.021s
W20260924 19:58:50.878800 20277 catalog_manager.cc:6038] Failed to send AlterTable RPC for tablet 65e0685bc4564703b88a5247511686ec (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 65e0685bc4564703b88a5247511686ec consensus state has no leader
I20260924 19:58:50.848858 20980 ts_tablet_manager.cc:1929] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:50.879369 20980 log.cc:1199] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/4b925a15a6074be0ae5d2c43a7ab83d1
I20260924 19:58:50.879289 20978 raft_consensus.cc:359] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.879906 20978 raft_consensus.cc:385] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.880146 20980 ts_tablet_manager.cc:1950] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:50.880177 20978 raft_consensus.cc:740] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.881538 20981 ts_tablet_manager.cc:1929] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:50.881912 20981 log.cc:1199] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/4b925a15a6074be0ae5d2c43a7ab83d1
I20260924 19:58:50.882637 20981 ts_tablet_manager.cc:1950] T 4b925a15a6074be0ae5d2c43a7ab83d1 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:50.886559 20978 consensus_queue.cc:260] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.888327 21140 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:58:50.891675 20982 raft_consensus.cc:2272] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:50.894712 20982 ts_tablet_manager.cc:1916] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:50.896431 20959 fs_manager.cc:730] Time spent opening block manager: real 0.067s user 0.004s sys 0.009s
I20260924 19:58:50.823920 20282 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 4b925a15a6074be0ae5d2c43a7ab83d1
I20260924 19:58:50.882748 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 4b925a15a6074be0ae5d2c43a7ab83d1 (table default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]) successfully deleted
I20260924 19:58:50.897042 20976 tablet_bootstrap.cc:492] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:50.899219 20700 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 with delete_type TABLET_DATA_DELETED (replaced by tablet b8ec997ee07b4953847c944ee1e9ec64) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:50.896867 20959 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data,/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal
uuid: "022c1898fcbe4856aa680c26047f6934"
format_stamp: "Formatted at 2026-09-24 19:58:34 on dist-test-slave-h376"
I20260924 19:58:50.899850 20959 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:58:50.901342 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 with delete_type TABLET_DATA_DELETED (replaced by tablet b8ec997ee07b4953847c944ee1e9ec64) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:50.902267 20977 tablet_bootstrap.cc:654] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:50.902740 20282 catalog_manager.cc:6934] ReplaceTablet: tablet 4b925a15a6074be0ae5d2c43a7ab83d1 deleted and replaced by tablet b8ec997ee07b4953847c944ee1e9ec64
I20260924 19:58:50.902858 20847 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 with delete_type TABLET_DATA_DELETED (replaced by tablet b8ec997ee07b4953847c944ee1e9ec64) from {username='slave'} at 127.0.0.1:60248
W20260924 19:58:50.904547 20271 catalog_manager.cc:5030] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 because the tablet was not found. No further retry: Not found: Tablet not found: 4b925a15a6074be0ae5d2c43a7ab83d1
I20260924 19:58:50.908965 20976 tablet_bootstrap.cc:654] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
W20260924 19:58:50.911275 20804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:33667, user_credentials={real_user=slave}} blocked reactor thread for 255828us
I20260924 19:58:50.825078 20280 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_1c56d0ae4f73481fab5eea13c17efd81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:50.917193 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8438dd32ff05426e947d9033981a3ea4" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:50.917850 20575 raft_consensus.cc:2468] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
I20260924 19:58:50.917810 20277 master_service.cc:960] ReplaceTablet: received request to replace tablet 635f4961769f49dbb908af7904d29042 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:50.921337 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8438dd32ff05426e947d9033981a3ea4" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:50.921977 20722 raft_consensus.cc:2468] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
I20260924 19:58:50.956667 20804 leader_election.cc:304] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters:
I20260924 19:58:50.961786 21130 raft_consensus.cc:2804] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:50.962155 21130 raft_consensus.cc:493] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:50.962458 21130 raft_consensus.cc:3060] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:50.962855 20977 tablet_bootstrap.cc:492] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:50.963269 20977 ts_tablet_manager.cc:1403] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.097s user 0.010s sys 0.006s
I20260924 19:58:50.970505 21130 raft_consensus.cc:515] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.972627 21130 leader_election.cc:290] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:50.965481 20977 raft_consensus.cc:359] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:50.975543 20977 raft_consensus.cc:385] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:50.975814 20977 raft_consensus.cc:740] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:50.976377 20977 consensus_queue.cc:260] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:50.981096 20283 catalog_manager.cc:5938] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601) does not have the latest schema for tablet 65e0685bc4564703b88a5247511686ec (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]). Expected version 1 got 0
I20260924 19:58:50.983692 21019 raft_consensus.cc:493] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:50.984148 21019 raft_consensus.cc:515] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:50.986083 21019 leader_election.cc:290] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:50.989749 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e0685bc4564703b88a5247511686ec" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:50.990319 20575 raft_consensus.cc:2468] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:50.990432 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e0685bc4564703b88a5247511686ec" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:50.990999 20867 raft_consensus.cc:2468] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:50.991611 20658 leader_election.cc:304] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:50.993040 20277 catalog_manager.cc:5938] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667) does not have the latest schema for tablet 65e0685bc4564703b88a5247511686ec (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]). Expected version 1 got 0
I20260924 19:58:50.994611 20977 ts_tablet_manager.cc:1434] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.031s user 0.004s sys 0.001s
I20260924 19:58:50.995834 20977 tablet_bootstrap.cc:492] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:50.996652 20283 catalog_manager.cc:5938] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601) does not have the latest schema for tablet 65e0685bc4564703b88a5247511686ec (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]). Expected version 1 got 0
I20260924 19:58:50.998394 21019 raft_consensus.cc:2804] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:50.998739 21019 raft_consensus.cc:493] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:50.999081 21019 raft_consensus.cc:3060] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.002388 20282 catalog_manager.cc:5938] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719) does not have the latest schema for tablet 8438dd32ff05426e947d9033981a3ea4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]). Expected version 1 got 0
I20260924 19:58:51.010001 21019 raft_consensus.cc:515] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.014710 20848 tablet_service.cc:1511] Processing CreateTablet for tablet 93f0ea69855145e5bcfd8080d2969dd0 (DEFAULT_TABLE table=default.table_6ac8d98b233146c495713e0cda0ac6b5 [id=46a1524a039b484d9118dee4b62cb8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:51.014663 20701 tablet_service.cc:1511] Processing CreateTablet for tablet 93f0ea69855145e5bcfd8080d2969dd0 (DEFAULT_TABLE table=default.table_6ac8d98b233146c495713e0cda0ac6b5 [id=46a1524a039b484d9118dee4b62cb8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:51.015672 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93f0ea69855145e5bcfd8080d2969dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:51.015936 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93f0ea69855145e5bcfd8080d2969dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:51.021086 20283 catalog_manager.cc:5938] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719) does not have the latest schema for tablet 65e0685bc4564703b88a5247511686ec (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]). Expected version 1 got 0
I20260924 19:58:51.026386 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 4b925a15a6074be0ae5d2c43a7ab83d1 (table default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]) successfully deleted
W20260924 19:58:51.027174 20271 catalog_manager.cc:5030] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 because the tablet was not found. No further retry: Not found: Tablet not found: 4b925a15a6074be0ae5d2c43a7ab83d1
I20260924 19:58:51.027518 20977 tablet_bootstrap.cc:654] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.028848 20553 tablet_service.cc:1511] Processing CreateTablet for tablet 93f0ea69855145e5bcfd8080d2969dd0 (DEFAULT_TABLE table=default.table_6ac8d98b233146c495713e0cda0ac6b5 [id=46a1524a039b484d9118dee4b62cb8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:51.029932 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93f0ea69855145e5bcfd8080d2969dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:51.037616 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e0685bc4564703b88a5247511686ec" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:51.037952 20554 tablet_service.cc:1511] Processing CreateTablet for tablet b8ec997ee07b4953847c944ee1e9ec64 (DEFAULT_TABLE table=default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:51.038184 20575 raft_consensus.cc:3060] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.039027 20554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ec997ee07b4953847c944ee1e9ec64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:51.044790 20575 raft_consensus.cc:2468] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:51.046038 20658 leader_election.cc:304] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:51.050191 20277 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_b9baeba47f2642b38898c5b2f045cccf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:51.051822 20977 tablet_bootstrap.cc:492] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:51.052660 20977 ts_tablet_manager.cc:1403] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.001s
I20260924 19:58:51.054153 20976 tablet_bootstrap.cc:492] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:51.054816 20976 ts_tablet_manager.cc:1403] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.158s user 0.006s sys 0.004s
I20260924 19:58:51.055022 20977 raft_consensus.cc:359] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.055660 20977 raft_consensus.cc:385] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.055920 20977 raft_consensus.cc:740] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:51.057122 20976 raft_consensus.cc:359] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.028100 20978 ts_tablet_manager.cc:1434] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.151s user 0.003s sys 0.004s
I20260924 19:58:51.061061 20977 consensus_queue.cc:260] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.061730 20702 tablet_service.cc:1511] Processing CreateTablet for tablet b8ec997ee07b4953847c944ee1e9ec64 (DEFAULT_TABLE table=default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:51.062731 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ec997ee07b4953847c944ee1e9ec64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:51.069161 20977 ts_tablet_manager.cc:1434] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260924 19:58:51.084822 20282 catalog_manager.cc:5938] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719) does not have the latest schema for tablet 65e0685bc4564703b88a5247511686ec (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]). Expected version 1 got 0
I20260924 19:58:51.089609 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8438dd32ff05426e947d9033981a3ea4" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:51.090595 20978 tablet_bootstrap.cc:492] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:51.093504 20846 tablet_service.cc:1511] Processing CreateTablet for tablet b8ec997ee07b4953847c944ee1e9ec64 (DEFAULT_TABLE table=default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:51.093755 20575 raft_consensus.cc:3060] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.094544 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ec997ee07b4953847c944ee1e9ec64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:51.095479 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65e0685bc4564703b88a5247511686ec" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:51.096004 20867 raft_consensus.cc:3060] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.102671 20982 ts_tablet_manager.cc:1929] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:51.093153 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8438dd32ff05426e947d9033981a3ea4" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
I20260924 19:58:51.104493 20722 raft_consensus.cc:3060] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.110510 20867 raft_consensus.cc:2468] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:51.112118 20722 raft_consensus.cc:2468] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 1.
I20260924 19:58:51.103343 21019 leader_election.cc:290] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:51.117175 20976 raft_consensus.cc:385] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.120673 21019 raft_consensus.cc:2804] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:51.124845 20976 raft_consensus.cc:740] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:51.126972 20976 consensus_queue.cc:260] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.133333 21019 raft_consensus.cc:697] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Becoming Leader. State: Replica: af20c71727a14e398a217d4623d1399a, State: Running, Role: LEADER
W20260924 19:58:51.126091 20658 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:33667, user_credentials={real_user=slave}} blocked reactor thread for 80103.6us
I20260924 19:58:51.118680 20804 leader_election.cc:304] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:51.147131 21130 raft_consensus.cc:2804] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:51.150657 21130 raft_consensus.cc:697] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: Becoming Leader. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Running, Role: LEADER
I20260924 19:58:51.151724 21130 consensus_queue.cc:237] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.155280 21019 consensus_queue.cc:237] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.156045 20276 catalog_manager.cc:2770] deleted HMS catalog entry for table default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]
I20260924 19:58:51.164430 20976 ts_tablet_manager.cc:1434] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.109s user 0.003s sys 0.002s
I20260924 19:58:51.167047 20976 tablet_bootstrap.cc:492] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:51.170529 20977 tablet_bootstrap.cc:492] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:51.103015 20982 log.cc:1199] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/4b925a15a6074be0ae5d2c43a7ab83d1
I20260924 19:58:51.176071 20977 tablet_bootstrap.cc:654] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.176417 20982 ts_tablet_manager.cc:1950] T 4b925a15a6074be0ae5d2c43a7ab83d1 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:51.180766 20575 raft_consensus.cc:2468] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 1.
I20260924 19:58:51.198032 20978 tablet_bootstrap.cc:654] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.197762 20282 catalog_manager.cc:5838] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a reported cstate change: term changed from 0 to 1, leader changed from <none> to af20c71727a14e398a217d4623d1399a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "af20c71727a14e398a217d4623d1399a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20260924 19:58:51.199535 20282 catalog_manager.cc:5938] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719) does not have the latest schema for tablet 65e0685bc4564703b88a5247511686ec (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]). Expected version 1 got 0
I20260924 19:58:51.210446 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 4b925a15a6074be0ae5d2c43a7ab83d1 (table default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]) successfully deleted
W20260924 19:58:51.211969 20270 catalog_manager.cc:5030] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): delete failed for tablet 4b925a15a6074be0ae5d2c43a7ab83d1 because the tablet was not found. No further retry: Not found: Tablet not found: 4b925a15a6074be0ae5d2c43a7ab83d1
I20260924 19:58:51.212730 20977 tablet_bootstrap.cc:492] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:51.225925 20977 ts_tablet_manager.cc:1403] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.056s user 0.011s sys 0.000s
I20260924 19:58:51.226092 20283 catalog_manager.cc:5838] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0055218dbcbd48b58018e9774e59e14c (127.17.2.131). New cstate: current_term: 1 leader_uuid: "0055218dbcbd48b58018e9774e59e14c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:51.227608 20283 catalog_manager.cc:5938] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601) does not have the latest schema for tablet 8438dd32ff05426e947d9033981a3ea4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]). Expected version 1 got 0
I20260924 19:58:51.228943 20977 raft_consensus.cc:359] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.229804 20977 raft_consensus.cc:385] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.230113 20977 raft_consensus.cc:740] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:51.231561 20959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260924 19:58:51.231232 20977 consensus_queue.cc:260] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.233618 20959 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:58:51.234194 20959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:58:51.251523 20977 ts_tablet_manager.cc:1434] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260924 19:58:51.253919 20976 tablet_bootstrap.cc:654] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.261361 20977 tablet_bootstrap.cc:492] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:51.266487 20978 tablet_bootstrap.cc:492] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:51.269393 21099 raft_consensus.cc:493] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:51.269824 21099 raft_consensus.cc:515] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.272876 21099 leader_election.cc:290] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:51.274585 20978 ts_tablet_manager.cc:1403] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.184s user 0.011s sys 0.000s
I20260924 19:58:51.274451 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2719ada60a143c6862e90d81f9c2d82" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:51.275662 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2719ada60a143c6862e90d81f9c2d82" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:51.276250 20867 raft_consensus.cc:2468] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 0.
I20260924 19:58:51.277164 20978 raft_consensus.cc:359] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:51.277801 20978 raft_consensus.cc:385] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.278045 20978 raft_consensus.cc:740] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
W20260924 19:58:51.278373 20509 leader_election.cc:343] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:51.279578 20508 leader_election.cc:304] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters: af20c71727a14e398a217d4623d1399a
I20260924 19:58:51.280588 21099 raft_consensus.cc:2804] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:51.280936 21099 raft_consensus.cc:493] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:51.281261 21099 raft_consensus.cc:3060] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.285075 20977 tablet_bootstrap.cc:654] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.278681 20978 consensus_queue.cc:260] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:51.292490 21099 raft_consensus.cc:515] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.295912 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2719ada60a143c6862e90d81f9c2d82" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
W20260924 19:58:51.297329 20509 leader_election.cc:343] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:51.298403 20976 tablet_bootstrap.cc:492] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:51.298837 20976 ts_tablet_manager.cc:1403] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.133s user 0.007s sys 0.002s
I20260924 19:58:51.299510 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2719ada60a143c6862e90d81f9c2d82" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:51.300041 20867 raft_consensus.cc:3060] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.301226 20976 raft_consensus.cc:359] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:51.301827 20976 raft_consensus.cc:385] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.302063 20976 raft_consensus.cc:740] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:51.302625 20978 ts_tablet_manager.cc:1434] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260924 19:58:51.309610 20978 tablet_bootstrap.cc:492] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:51.315200 20978 tablet_bootstrap.cc:654] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.323848 20976 consensus_queue.cc:260] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:51.327917 20867 raft_consensus.cc:2468] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 1.
I20260924 19:58:51.329367 20508 leader_election.cc:304] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters: af20c71727a14e398a217d4623d1399a
I20260924 19:58:51.333735 20976 ts_tablet_manager.cc:1434] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.035s user 0.001s sys 0.005s
I20260924 19:58:51.335302 21159 raft_consensus.cc:2804] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:51.345316 20976 tablet_bootstrap.cc:492] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:51.349915 20977 tablet_bootstrap.cc:492] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:51.350419 21099 leader_election.cc:290] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Requested vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:51.350419 20977 ts_tablet_manager.cc:1403] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.089s user 0.010s sys 0.000s
I20260924 19:58:51.353641 20977 raft_consensus.cc:359] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:51.354283 20977 raft_consensus.cc:385] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.354570 20977 raft_consensus.cc:740] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:51.355263 21159 raft_consensus.cc:697] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: Becoming Leader. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Running, Role: LEADER
I20260924 19:58:51.356133 21159 consensus_queue.cc:237] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.367975 20976 tablet_bootstrap.cc:654] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.370623 20977 consensus_queue.cc:260] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:51.373054 20977 ts_tablet_manager.cc:1434] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260924 19:58:51.378872 21161 raft_consensus.cc:493] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:51.379410 21161 raft_consensus.cc:515] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.382470 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ec997ee07b4953847c944ee1e9ec64" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
W20260924 19:58:51.384473 20509 leader_election.cc:343] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:51.386636 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ec997ee07b4953847c944ee1e9ec64" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
W20260924 19:58:51.388039 20508 leader_election.cc:343] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:51.393033 20868 raft_consensus.cc:1275] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Refusing update from remote peer af20c71727a14e398a217d4623d1399a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:58:51.394795 21145 consensus_queue.cc:1048] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:58:51.393401 20575 raft_consensus.cc:1275] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Refusing update from remote peer af20c71727a14e398a217d4623d1399a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:58:51.403354 21145 consensus_queue.cc:1048] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:51.409291 20283 master_service.cc:960] ReplaceTablet: received request to replace tablet e01e0614ba6e4404a393ed67b6499c4b from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:51.412139 20978 tablet_bootstrap.cc:492] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:51.412616 20978 ts_tablet_manager.cc:1403] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.001s
I20260924 19:58:51.413183 20508 leader_election.cc:304] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:58:51.418457 21099 raft_consensus.cc:2749] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:51.426573 21161 leader_election.cc:290] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:58:51.437047 20508 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.131:41601, user_credentials={real_user=slave}} blocked reactor thread for 49013.2us
I20260924 19:58:51.437518 20976 tablet_bootstrap.cc:492] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:51.438032 20976 ts_tablet_manager.cc:1403] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.005s
I20260924 19:58:51.440639 20976 raft_consensus.cc:359] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:51.441253 20976 raft_consensus.cc:385] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.441498 20976 raft_consensus.cc:740] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:51.443567 20978 raft_consensus.cc:359] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.444213 20978 raft_consensus.cc:385] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.444459 20978 raft_consensus.cc:740] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:51.450344 21174 raft_consensus.cc:493] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:51.451181 21174 raft_consensus.cc:515] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:51.451713 20978 consensus_queue.cc:260] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.454725 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93f0ea69855145e5bcfd8080d2969dd0" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:51.463480 20978 ts_tablet_manager.cc:1434] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.051s user 0.005s sys 0.003s
W20260924 19:58:51.464726 20509 leader_election.cc:343] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:51.468209 21157 mvcc.cc:204] Tried to move back new op lower bound from 7332986598962712576 to 7332986598025535488. Current Snapshot: MvccSnapshot[applied={T|T < 7332986598962712576}]
I20260924 19:58:51.473026 20978 tablet_bootstrap.cc:492] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:51.472293 21171 mvcc.cc:204] Tried to move back new op lower bound from 7332986598962712576 to 7332986598025535488. Current Snapshot: MvccSnapshot[applied={T|T < 7332986598962712576}]
I20260924 19:58:51.481645 21174 leader_election.cc:290] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:51.483350 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93f0ea69855145e5bcfd8080d2969dd0" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:51.491838 21184 tablet.cc:1751] T 65e0685bc4564703b88a5247511686ec P 0055218dbcbd48b58018e9774e59e14c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:58:51.503991 21183 tablet.cc:1751] T 65e0685bc4564703b88a5247511686ec P af20c71727a14e398a217d4623d1399a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:58:51.505044 20283 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet e01e0614ba6e4404a393ed67b6499c4b
I20260924 19:58:51.506090 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet e01e0614ba6e4404a393ed67b6499c4b with delete_type TABLET_DATA_DELETED (replaced by tablet 67bcc13a05734fe4abfd011d603d31ad) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:51.507102 20981 tablet_replica.cc:333] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:51.507792 20981 raft_consensus.cc:2243] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:51.508224 20981 raft_consensus.cc:2272] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:51.515130 20981 ts_tablet_manager.cc:1916] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:51.515915 20978 tablet_bootstrap.cc:654] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.517014 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet e01e0614ba6e4404a393ed67b6499c4b with delete_type TABLET_DATA_DELETED (replaced by tablet 67bcc13a05734fe4abfd011d603d31ad) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:51.517719 20982 tablet_replica.cc:333] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:51.518411 20982 raft_consensus.cc:2243] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:51.518822 20982 raft_consensus.cc:2272] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:51.521111 20982 ts_tablet_manager.cc:1916] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:51.544243 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet e01e0614ba6e4404a393ed67b6499c4b with delete_type TABLET_DATA_DELETED (replaced by tablet 67bcc13a05734fe4abfd011d603d31ad) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:51.548305 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet e01e0614ba6e4404a393ed67b6499c4b with delete_type TABLET_DATA_DELETED (replaced by tablet 67bcc13a05734fe4abfd011d603d31ad) from {username='slave'} at 127.0.0.1:48342
W20260924 19:58:51.553016 20271 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 62f4d38a8da14252982bcac0e7df28c4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 62f4d38a8da14252982bcac0e7df28c4 consensus state has no leader
I20260924 19:58:51.554190 20978 tablet_bootstrap.cc:492] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:51.554615 20978 ts_tablet_manager.cc:1403] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.008s
I20260924 19:58:51.556890 20283 catalog_manager.cc:6934] ReplaceTablet: tablet e01e0614ba6e4404a393ed67b6499c4b deleted and replaced by tablet 67bcc13a05734fe4abfd011d603d31ad
I20260924 19:58:51.457669 20976 consensus_queue.cc:260] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:51.563621 20867 raft_consensus.cc:2468] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 0.
I20260924 19:58:51.564991 20978 raft_consensus.cc:359] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.565768 20978 raft_consensus.cc:385] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.566007 20978 raft_consensus.cc:740] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:51.568006 21173 mvcc.cc:204] Tried to move back new op lower bound from 7332986598962712576 to 7332986598025535488. Current Snapshot: MvccSnapshot[applied={T|T < 7332986598962712576}]
I20260924 19:58:51.568821 20508 leader_election.cc:304] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters: af20c71727a14e398a217d4623d1399a
I20260924 19:58:51.572391 21159 raft_consensus.cc:2804] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:51.572734 21159 raft_consensus.cc:493] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:51.573053 21159 raft_consensus.cc:3060] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.575619 20976 ts_tablet_manager.cc:1434] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.137s user 0.006s sys 0.000s
I20260924 19:58:51.557904 20282 catalog_manager.cc:5838] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "30a4f50fc28c4616ba622ba51b0b2622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20260924 19:58:51.566583 20978 consensus_queue.cc:260] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.583958 21193 tablet_replica.cc:333] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:51.585911 21193 raft_consensus.cc:2243] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:51.586519 21193 raft_consensus.cc:2272] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:51.589519 20976 tablet_bootstrap.cc:492] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:51.594362 20283 catalog_manager.cc:5938] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601) does not have the latest schema for tablet 65e0685bc4564703b88a5247511686ec (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0]). Expected version 1 got 0
I20260924 19:58:51.617309 21159 raft_consensus.cc:515] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:51.620260 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93f0ea69855145e5bcfd8080d2969dd0" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
W20260924 19:58:51.621636 20509 leader_election.cc:343] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:51.625588 21198 tablet.cc:1751] T 65e0685bc4564703b88a5247511686ec P 30a4f50fc28c4616ba622ba51b0b2622: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:58:51.637010 20978 ts_tablet_manager.cc:1434] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.082s user 0.014s sys 0.005s
I20260924 19:58:51.638837 20978 tablet_bootstrap.cc:492] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:51.644243 20981 ts_tablet_manager.cc:1929] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:51.644625 20981 log.cc:1199] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/e01e0614ba6e4404a393ed67b6499c4b
I20260924 19:58:51.645241 20981 ts_tablet_manager.cc:1950] T e01e0614ba6e4404a393ed67b6499c4b P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:51.650848 21159 leader_election.cc:290] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Requested vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:51.646561 20978 tablet_bootstrap.cc:654] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.666424 21193 ts_tablet_manager.cc:1916] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:51.675129 20978 tablet_bootstrap.cc:492] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:51.675562 20978 ts_tablet_manager.cc:1403] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.002s
I20260924 19:58:51.678030 20978 raft_consensus.cc:359] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:51.678608 20978 raft_consensus.cc:385] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.678841 20978 raft_consensus.cc:740] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:51.687901 20978 consensus_queue.cc:260] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:51.691480 20978 ts_tablet_manager.cc:1434] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.016s user 0.003s sys 0.002s
I20260924 19:58:51.715005 20554 tablet_service.cc:1511] Processing CreateTablet for tablet 67bcc13a05734fe4abfd011d603d31ad (DEFAULT_TABLE table=default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:51.716337 20554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67bcc13a05734fe4abfd011d603d31ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:51.719223 20702 tablet_service.cc:1511] Processing CreateTablet for tablet 67bcc13a05734fe4abfd011d603d31ad (DEFAULT_TABLE table=default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:51.720278 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67bcc13a05734fe4abfd011d603d31ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:51.721904 20982 ts_tablet_manager.cc:1929] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:51.722050 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93f0ea69855145e5bcfd8080d2969dd0" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:51.722270 20982 log.cc:1199] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/e01e0614ba6e4404a393ed67b6499c4b
I20260924 19:58:51.722585 20867 raft_consensus.cc:3060] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.722910 20982 ts_tablet_manager.cc:1950] T e01e0614ba6e4404a393ed67b6499c4b P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:51.726589 20976 tablet_bootstrap.cc:654] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.731635 20846 tablet_service.cc:1511] Processing CreateTablet for tablet 67bcc13a05734fe4abfd011d603d31ad (DEFAULT_TABLE table=default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:51.731937 21201 raft_consensus.cc:493] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:51.732635 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67bcc13a05734fe4abfd011d603d31ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:51.732409 21201 raft_consensus.cc:515] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.734426 21201 leader_election.cc:290] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:51.735908 20868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ec997ee07b4953847c944ee1e9ec64" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:51.736856 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ec997ee07b4953847c944ee1e9ec64" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:51.737470 20574 raft_consensus.cc:2468] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
W20260924 19:58:51.738670 20657 leader_election.cc:343] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:51.739827 20658 leader_election.cc:304] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c
I20260924 19:58:51.741019 21201 raft_consensus.cc:2804] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:51.741423 21201 raft_consensus.cc:493] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:51.741806 21201 raft_consensus.cc:3060] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.747836 20867 raft_consensus.cc:2468] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 1.
I20260924 19:58:51.712383 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet e01e0614ba6e4404a393ed67b6499c4b (table default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]) successfully deleted
W20260924 19:58:51.749048 20271 catalog_manager.cc:5030] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet e01e0614ba6e4404a393ed67b6499c4b because the tablet was not found. No further retry: Not found: Tablet not found: e01e0614ba6e4404a393ed67b6499c4b
I20260924 19:58:51.766216 21201 raft_consensus.cc:515] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.771293 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet e01e0614ba6e4404a393ed67b6499c4b (table default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]) successfully deleted
I20260924 19:58:51.779040 20722 raft_consensus.cc:1275] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Refusing update from remote peer 0055218dbcbd48b58018e9774e59e14c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:58:51.780221 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ec997ee07b4953847c944ee1e9ec64" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:51.780776 20575 raft_consensus.cc:3060] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.787801 20575 raft_consensus.cc:2468] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:51.789062 20658 leader_election.cc:304] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:51.789899 21145 raft_consensus.cc:2804] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:51.790315 21145 raft_consensus.cc:697] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Becoming Leader. State: Replica: af20c71727a14e398a217d4623d1399a, State: Running, Role: LEADER
I20260924 19:58:51.790648 20976 tablet_bootstrap.cc:492] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:51.791047 20976 ts_tablet_manager.cc:1403] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.202s user 0.011s sys 0.002s
I20260924 19:58:51.791064 21145 consensus_queue.cc:237] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.793531 20976 raft_consensus.cc:359] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.800021 20508 leader_election.cc:304] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters: af20c71727a14e398a217d4623d1399a
I20260924 19:58:51.801041 21159 raft_consensus.cc:2804] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:51.801472 21159 raft_consensus.cc:697] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: Becoming Leader. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Running, Role: LEADER
I20260924 19:58:51.802183 21159 consensus_queue.cc:237] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:51.813387 20283 catalog_manager.cc:6391] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0] alter complete (version 1)
I20260924 19:58:51.814733 21201 leader_election.cc:290] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:51.815989 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8ec997ee07b4953847c944ee1e9ec64" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:51.816977 20575 raft_consensus.cc:1275] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Refusing update from remote peer 0055218dbcbd48b58018e9774e59e14c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:58:51.820896 21212 consensus_queue.cc:1048] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260924 19:58:51.794139 20976 raft_consensus.cc:385] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.828186 20976 raft_consensus.cc:740] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:51.836042 20976 consensus_queue.cc:260] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.836861 20867 raft_consensus.cc:3060] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.830160 21145 raft_consensus.cc:493] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:51.839326 21145 raft_consensus.cc:515] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.840080 21130 consensus_queue.cc:1048] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:51.857105 20867 raft_consensus.cc:2468] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:51.862978 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2719ada60a143c6862e90d81f9c2d82" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:51.864250 21193 ts_tablet_manager.cc:1929] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:51.866180 20976 ts_tablet_manager.cc:1434] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.075s user 0.005s sys 0.001s
I20260924 19:58:51.867866 21145 leader_election.cc:290] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:51.880126 20978 tablet_bootstrap.cc:492] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:51.880792 20275 catalog_manager.cc:2213] created HMS catalog entry for table [id=d9d37737d55c4be39d471fcf624a8866]
I20260924 19:58:51.882508 21193 log.cc:1199] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/e01e0614ba6e4404a393ed67b6499c4b
I20260924 19:58:51.868842 20868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2719ada60a143c6862e90d81f9c2d82" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:51.884956 20868 raft_consensus.cc:2393] T e2719ada60a143c6862e90d81f9c2d82 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af20c71727a14e398a217d4623d1399a in current term 1: Already voted for candidate 30a4f50fc28c4616ba622ba51b0b2622 in this term.
I20260924 19:58:51.886040 20657 leader_election.cc:304] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:58:51.887240 21201 raft_consensus.cc:3060] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:51.888530 20978 tablet_bootstrap.cc:654] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.890022 21193 ts_tablet_manager.cc:1950] T e01e0614ba6e4404a393ed67b6499c4b P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:51.846180 20282 catalog_manager.cc:5838] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a reported cstate change: term changed from 0 to 1, leader changed from <none> to af20c71727a14e398a217d4623d1399a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "af20c71727a14e398a217d4623d1399a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: UNKNOWN } } }
I20260924 19:58:51.901211 20275 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 52) took 3016 ms. Trace:
I20260924 19:58:51.904467 20976 tablet_bootstrap.cc:492] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:51.901693 20275 rpcz_store.cc:276] 0924 19:58:48.884529 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:48.887834 (+ 3305us) service_pool.cc:224] Handling call
0924 19:58:49.870827 (+982993us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:49.872657 (+ 1830us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:51.880612 (+2007955us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:51.900254 (+ 19642us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:58:51.900608 (+ 354us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:58:51.901115 (+ 507us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:51.882802 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:51.883212 (+ 410us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:51.883235 (+ 23us) write_op.cc:435] Acquired schema lock
0924 19:58:51.883247 (+ 12us) tablet.cc:662] Decoding operations
0924 19:58:51.883683 (+ 436us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:51.883723 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:51.883744 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:51.883946 (+ 202us) tablet.cc:701] Row locks acquired
0924 19:58:51.883960 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:58:51.884069 (+ 109us) write_op.cc:270] Start()
0924 19:58:51.884146 (+ 77us) write_op.cc:276] Timestamp: P: 1790279931883987 usec, L: 0
0924 19:58:51.884161 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:58:51.884646 (+ 485us) log.cc:844] Serialized 432 byte log entry
0924 19:58:51.897377 (+ 12731us) op_driver.cc:464] REPLICATION: finished
0924 19:58:51.897733 (+ 356us) write_op.cc:301] APPLY: starting
0924 19:58:51.897791 (+ 58us) tablet.cc:1394] starting BulkCheckPresence
0924 19:58:51.897909 (+ 118us) tablet.cc:1397] finished BulkCheckPresence
0924 19:58:51.898012 (+ 103us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:58:51.898446 (+ 434us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:58:51.898606 (+ 160us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:58:51.898649 (+ 43us) write_op.cc:312] APPLY: finished
0924 19:58:51.898832 (+ 183us) log.cc:844] Serialized 28 byte log entry
0924 19:58:51.898973 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:51.899039 (+ 66us) write_op.cc:454] Released schema lock
0924 19:58:51.899073 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0924 19:58:51.899320 (+ 247us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:51.899341 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1281791,"Hive Metastore.run_cpu_time_us":872,"Hive Metastore.run_wall_time_us":724754,"child_traces":[["op",{"apply.queue_time_us":184,"apply.run_cpu_time_us":1564,"apply.run_wall_time_us":1558,"num_ops":2,"prepare.queue_time_us":173,"prepare.run_cpu_time_us":2146,"prepare.run_wall_time_us":2147,"raft.queue_time_us":451,"raft.run_cpu_time_us":822,"raft.run_wall_time_us":9126,"replication_time_us":13104}]]}
I20260924 19:58:51.914578 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet e01e0614ba6e4404a393ed67b6499c4b (table default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]) successfully deleted
I20260924 19:58:51.916576 20283 master_service.cc:960] ReplaceTablet: received request to replace tablet e01e0614ba6e4404a393ed67b6499c4b from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:51.922135 20976 tablet_bootstrap.cc:654] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:51.936964 20978 tablet_bootstrap.cc:492] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:51.937836 20978 ts_tablet_manager.cc:1403] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.005s
I20260924 19:58:51.941164 20978 raft_consensus.cc:359] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.943313 21227 tablet_bootstrap.cc:492] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:51.919142 20283 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c40348984314595b6e6e49c4f03f35e from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:51.952927 21184 tablet.cc:1751] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:58:51.965907 20978 raft_consensus.cc:385] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:51.968279 20978 raft_consensus.cc:740] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:51.971797 20275 catalog_manager.cc:5838] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "30a4f50fc28c4616ba622ba51b0b2622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:51.978852 21198 tablet.cc:1751] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:58:51.985105 20702 tablet_service.cc:1511] Processing CreateTablet for tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7 (DEFAULT_TABLE table=default.table_7a523607282c4d1c85c7a42deabb470b [id=d9d37737d55c4be39d471fcf624a8866]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:51.987568 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:51.987741 20976 tablet_bootstrap.cc:492] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:51.988119 20976 ts_tablet_manager.cc:1403] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.084s user 0.010s sys 0.000s
I20260924 19:58:51.989943 21183 tablet.cc:1751] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260924 19:58:51.990491 20976 raft_consensus.cc:359] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:51.999692 20978 consensus_queue.cc:260] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:52.002489 21235 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.17.2.188:35311 every 8 connection(s)
I20260924 19:58:52.004914 20846 tablet_service.cc:1511] Processing CreateTablet for tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7 (DEFAULT_TABLE table=default.table_7a523607282c4d1c85c7a42deabb470b [id=d9d37737d55c4be39d471fcf624a8866]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:52.005946 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:51.991065 20976 raft_consensus.cc:385] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:52.009349 21227 tablet_bootstrap.cc:654] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:52.009611 20959 rpc_server.cc:307] RPC server started. Bound to: 127.17.2.188:35311
I20260924 19:58:52.012634 20976 raft_consensus.cc:740] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:52.013336 20978 ts_tablet_manager.cc:1434] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.075s user 0.006s sys 0.001s
I20260924 19:58:52.013664 20959 server_base.cc:1193] Dumped server information to /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/master-2/data/info.pb
I20260924 19:58:52.016307 20554 tablet_service.cc:1511] Processing CreateTablet for tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7 (DEFAULT_TABLE table=default.table_7a523607282c4d1c85c7a42deabb470b [id=d9d37737d55c4be39d471fcf624a8866]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:52.016410 20976 consensus_queue.cc:260] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:52.017446 20554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:52.022805 17418 external_mini_cluster.cc:1434] Started /tmp/dist-test-task0eKU8U/build/tsan/bin/kudu as pid 20959
I20260924 19:58:52.024874 20976 ts_tablet_manager.cc:1434] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20260924 19:58:52.063824 20978 tablet_bootstrap.cc:492] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:52.068645 21237 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:58:52.069650 20978 tablet_bootstrap.cc:654] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:52.080309 21227 tablet_bootstrap.cc:492] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:52.080899 21227 ts_tablet_manager.cc:1403] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.138s user 0.014s sys 0.000s
W20260924 19:58:52.081708 21219 log.cc:927] Time spent T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20260924 19:58:52.083076 20913 heartbeater.cc:344] Connected to a master server at 127.17.2.188:35311
I20260924 19:58:52.083623 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:52.083761 21227 raft_consensus.cc:359] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:52.084460 21227 raft_consensus.cc:385] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:52.084725 21227 raft_consensus.cc:740] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:52.086793 21230 raft_consensus.cc:493] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:52.087208 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: Bootstrap starting.
I20260924 19:58:52.087272 21230 raft_consensus.cc:515] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:52.087961 20767 heartbeater.cc:344] Connected to a master server at 127.17.2.188:35311
I20260924 19:58:52.089349 21230 leader_election.cc:290] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:52.093943 21227 consensus_queue.cc:260] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:52.101303 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:52.106098 20284 catalog_manager.cc:2213] created HMS catalog entry for table [id=99cc6ce05d86411d93e5afc71ae06fb2]
I20260924 19:58:52.106752 21237 log.cc:826] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: Log is configured to *not* fsync() on all Append() calls
I20260924 19:58:52.111459 20282 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_9067ff1ac3de431993245b7bafa4a8df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:52.124673 20275 catalog_manager.cc:6391] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29] alter complete (version 1)
I20260924 19:58:52.143428 20620 heartbeater.cc:344] Connected to a master server at 127.17.2.188:35311
I20260924 19:58:52.147212 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:58:52.148276 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: Bootstrap complete.
I20260924 19:58:52.150856 20978 tablet_bootstrap.cc:492] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:52.151345 20978 ts_tablet_manager.cc:1403] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.088s user 0.005s sys 0.005s
I20260924 19:58:52.161898 20438 catalog_manager.cc:2811] Deleting table default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0] in response to Hive Metastore notification log event 12
I20260924 19:58:52.167565 20978 raft_consensus.cc:359] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.168251 20978 raft_consensus.cc:385] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:52.168503 20978 raft_consensus.cc:740] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:52.169363 21227 ts_tablet_manager.cc:1434] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.088s user 0.006s sys 0.000s
I20260924 19:58:52.170037 20284 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 58) took 2726 ms. Trace:
I20260924 19:58:52.170380 20284 rpcz_store.cc:276] 0924 19:58:49.443972 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:49.451936 (+ 7964us) service_pool.cc:224] Handling call
0924 19:58:49.875369 (+423433us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:49.877191 (+ 1822us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:52.105946 (+2228755us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:52.169193 (+ 63247us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:58:52.169522 (+ 329us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:58:52.169972 (+ 450us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:52.110248 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:52.110657 (+ 409us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:52.110682 (+ 25us) write_op.cc:435] Acquired schema lock
0924 19:58:52.110694 (+ 12us) tablet.cc:662] Decoding operations
0924 19:58:52.110966 (+ 272us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:52.111003 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:52.111023 (+ 20us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:52.111268 (+ 245us) tablet.cc:701] Row locks acquired
0924 19:58:52.111281 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:58:52.111393 (+ 112us) write_op.cc:270] Start()
0924 19:58:52.111469 (+ 76us) write_op.cc:276] Timestamp: P: 1790279932111300 usec, L: 0
0924 19:58:52.111483 (+ 14us) op_driver.cc:348] REPLICATION: starting
0924 19:58:52.111979 (+ 496us) log.cc:844] Serialized 432 byte log entry
0924 19:58:52.120108 (+ 8129us) op_driver.cc:464] REPLICATION: finished
0924 19:58:52.166991 (+ 46883us) write_op.cc:301] APPLY: starting
0924 19:58:52.167048 (+ 57us) tablet.cc:1394] starting BulkCheckPresence
0924 19:58:52.167173 (+ 125us) tablet.cc:1397] finished BulkCheckPresence
0924 19:58:52.167273 (+ 100us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:58:52.167635 (+ 362us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:58:52.167766 (+ 131us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:58:52.167807 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:58:52.167997 (+ 190us) log.cc:844] Serialized 28 byte log entry
0924 19:58:52.168135 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:52.168373 (+ 238us) write_op.cc:454] Released schema lock
0924 19:58:52.168408 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0924 19:58:52.168674 (+ 266us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:52.168697 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":2002718,"Hive Metastore.run_cpu_time_us":857,"Hive Metastore.run_wall_time_us":224877,"child_traces":[["op",{"apply.queue_time_us":46718,"apply.run_cpu_time_us":1432,"apply.run_wall_time_us":1647,"num_ops":2,"prepare.queue_time_us":2282,"prepare.run_cpu_time_us":2045,"prepare.run_wall_time_us":2051,"raft.queue_time_us":523,"raft.run_cpu_time_us":709,"raft.run_wall_time_us":712,"replication_time_us":8511}]]}
I20260924 19:58:52.177825 21227 tablet_bootstrap.cc:492] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:52.177795 20284 master_service.cc:960] ReplaceTablet: received request to replace tablet b8ec997ee07b4953847c944ee1e9ec64 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:52.183429 20978 consensus_queue.cc:260] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.184525 21145 raft_consensus.cc:493] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:52.185004 21145 raft_consensus.cc:515] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
W20260924 19:58:52.190506 20657 leader_election.cc:343] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 58e9ed4d1fee4d3ca2aa1df1e8d645f7
I20260924 19:58:52.183303 21237 raft_consensus.cc:359] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:52.199797 21237 raft_consensus.cc:740] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022c1898fcbe4856aa680c26047f6934, State: Initialized, Role: FOLLOWER
I20260924 19:58:52.200975 21237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:52.185377 20978 ts_tablet_manager.cc:1434] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.034s user 0.004s sys 0.000s
W20260924 19:58:52.210490 21183 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.220s user 0.001s sys 0.000s
I20260924 19:58:52.212379 21227 tablet_bootstrap.cc:654] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:52.214735 21244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:52.215662 21244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:52.218405 21237 sys_catalog.cc:565] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:58:52.226001 21198 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.246s user 0.001s sys 0.000s
I20260924 19:58:52.245321 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67bcc13a05734fe4abfd011d603d31ad" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:52.245572 21227 tablet_bootstrap.cc:492] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:52.246024 21227 ts_tablet_manager.cc:1403] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.000s
I20260924 19:58:52.245935 20575 raft_consensus.cc:2468] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
I20260924 19:58:52.249809 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67bcc13a05734fe4abfd011d603d31ad" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:52.249943 21227 raft_consensus.cc:359] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.250634 21227 raft_consensus.cc:385] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:52.250519 20722 raft_consensus.cc:2468] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
I20260924 19:58:52.250946 21227 raft_consensus.cc:740] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:52.268119 21227 consensus_queue.cc:260] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.272665 20284 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c40348984314595b6e6e49c4f03f35e from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:52.274463 20438 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet b8ec997ee07b4953847c944ee1e9ec64
I20260924 19:58:52.276304 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet b8ec997ee07b4953847c944ee1e9ec64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:52 UTC) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:52.277029 21193 tablet_replica.cc:333] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:52.277784 21193 raft_consensus.cc:2243] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:52.279870 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet b8ec997ee07b4953847c944ee1e9ec64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:52 UTC) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:52.280748 21193 raft_consensus.cc:2272] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:52.282735 21193 ts_tablet_manager.cc:1916] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:52.283378 21227 ts_tablet_manager.cc:1434] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.036s user 0.002s sys 0.005s
I20260924 19:58:52.343351 21252 tablet_replica.cc:333] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:52.356771 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58e9ed4d1fee4d3ca2aa1df1e8d645f7" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:52.357375 20575 raft_consensus.cc:2468] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:52.380484 20284 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_5edadeb38f0143aea238c6775f09b6e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:52.355876 21145 leader_election.cc:290] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601), 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667)
I20260924 19:58:52.262797 21201 raft_consensus.cc:2749] T e2719ada60a143c6862e90d81f9c2d82 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:52.358549 20658 leader_election.cc:304] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c
I20260924 19:58:52.385023 21252 raft_consensus.cc:2243] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:52.385725 21252 pending_rounds.cc:70] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: Trying to abort 1 pending ops.
I20260924 19:58:52.385973 21252 pending_rounds.cc:77] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986600625422336 op_type: NO_OP noop_request { }
I20260924 19:58:52.386831 21252 raft_consensus.cc:2889] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:58:52.387199 21252 raft_consensus.cc:2272] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:52.389696 21252 ts_tablet_manager.cc:1916] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:52.401911 21201 raft_consensus.cc:2804] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:52.413637 21201 raft_consensus.cc:493] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:52.414165 21201 raft_consensus.cc:3060] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:52.419435 20804 leader_election.cc:304] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:52.424435 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet b8ec997ee07b4953847c944ee1e9ec64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:52 UTC) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:52.429600 21130 raft_consensus.cc:2804] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:52.430058 21130 raft_consensus.cc:493] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:52.430374 21130 raft_consensus.cc:3060] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:52.437815 20976 tablet_bootstrap.cc:492] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:52.452863 20976 tablet_bootstrap.cc:654] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:52.454684 21159 consensus_queue.cc:1048] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:52.455914 21130 raft_consensus.cc:515] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:52.461300 21130 leader_election.cc:290] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:52.465849 20702 tablet_service.cc:1511] Processing CreateTablet for tablet c54e96fe673f438eb2919be2ad785641 (DEFAULT_TABLE table=default.table_2a7727ff525b44bc97232bb07f824105 [id=99cc6ce05d86411d93e5afc71ae06fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:52.466934 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c54e96fe673f438eb2919be2ad785641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:52.480611 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:52.485837 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67bcc13a05734fe4abfd011d603d31ad" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:52.486509 20575 raft_consensus.cc:3060] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:52.488905 20976 tablet_bootstrap.cc:492] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:52.489315 20976 ts_tablet_manager.cc:1403] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.004s
I20260924 19:58:52.492982 21194 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:58:52.494026 20575 raft_consensus.cc:2468] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 1.
I20260924 19:58:52.504953 21193 ts_tablet_manager.cc:1929] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:52.510959 21255 tablet_replica.cc:333] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:52.512192 21255 raft_consensus.cc:2243] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:52.512848 21255 raft_consensus.cc:2272] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:52.515089 21255 ts_tablet_manager.cc:1916] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:52.515925 21193 log.cc:1199] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/b8ec997ee07b4953847c944ee1e9ec64
I20260924 19:58:52.516783 21193 ts_tablet_manager.cc:1950] T b8ec997ee07b4953847c944ee1e9ec64 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:52.526024 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet b8ec997ee07b4953847c944ee1e9ec64 (table default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]) successfully deleted
I20260924 19:58:52.536317 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "67bcc13a05734fe4abfd011d603d31ad" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
I20260924 19:58:52.536927 20722 raft_consensus.cc:3060] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:52.543624 20722 raft_consensus.cc:2468] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 1.
I20260924 19:58:52.572783 20275 master_service.cc:960] ReplaceTablet: received request to replace tablet 8438dd32ff05426e947d9033981a3ea4 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:52.582165 20554 tablet_service.cc:1511] Processing CreateTablet for tablet c54e96fe673f438eb2919be2ad785641 (DEFAULT_TABLE table=default.table_2a7727ff525b44bc97232bb07f824105 [id=99cc6ce05d86411d93e5afc71ae06fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:52.583278 20554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c54e96fe673f438eb2919be2ad785641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:52.613389 21252 ts_tablet_manager.cc:1929] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:52.620530 21265 consensus_queue.cc:1048] T e2719ada60a143c6862e90d81f9c2d82 P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:52.623344 21252 log.cc:1199] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/b8ec997ee07b4953847c944ee1e9ec64
I20260924 19:58:52.624271 21252 ts_tablet_manager.cc:1950] T b8ec997ee07b4953847c944ee1e9ec64 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:52.627327 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet b8ec997ee07b4953847c944ee1e9ec64 (table default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]) successfully deleted
I20260924 19:58:52.630659 20275 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 8438dd32ff05426e947d9033981a3ea4
I20260924 19:58:52.632295 20275 catalog_manager.cc:6934] ReplaceTablet: tablet 8438dd32ff05426e947d9033981a3ea4 deleted and replaced by tablet e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:58:52.632997 20275 master_service.cc:960] ReplaceTablet: received request to replace tablet 67bcc13a05734fe4abfd011d603d31ad from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:52.636813 20701 tablet_service.cc:1558] Processing DeleteTablet for tablet 8438dd32ff05426e947d9033981a3ea4 with delete_type TABLET_DATA_DELETED (replaced by tablet e6dcbdac79d04942b3ac59dc11a0ed3c) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:52.638464 21145 raft_consensus.cc:493] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:52.638949 21145 raft_consensus.cc:515] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.641032 21145 leader_election.cc:290] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601), 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667)
I20260924 19:58:52.639729 21201 raft_consensus.cc:515] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.644570 20553 tablet_service.cc:1558] Processing DeleteTablet for tablet 8438dd32ff05426e947d9033981a3ea4 with delete_type TABLET_DATA_DELETED (replaced by tablet e6dcbdac79d04942b3ac59dc11a0ed3c) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:52.645272 21193 tablet_replica.cc:333] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:52.646062 21193 raft_consensus.cc:2243] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:52.646624 21193 raft_consensus.cc:2272] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:52.649394 21193 ts_tablet_manager.cc:1916] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:52.516963 20976 raft_consensus.cc:359] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.563665 20848 tablet_service.cc:1511] Processing CreateTablet for tablet c54e96fe673f438eb2919be2ad785641 (DEFAULT_TABLE table=default.table_2a7727ff525b44bc97232bb07f824105 [id=99cc6ce05d86411d93e5afc71ae06fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:52.656459 20976 raft_consensus.cc:385] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:52.656757 20976 raft_consensus.cc:740] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:52.657182 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c54e96fe673f438eb2919be2ad785641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:52.657373 20976 consensus_queue.cc:260] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.659334 20976 ts_tablet_manager.cc:1434] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.170s user 0.007s sys 0.001s
I20260924 19:58:52.660081 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:52.668665 21201 leader_election.cc:290] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Requested vote from peers 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601), 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667)
I20260924 19:58:52.683712 20575 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93f0ea69855145e5bcfd8080d2969dd0" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:52.684567 21227 tablet_bootstrap.cc:492] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:52.691078 21252 tablet_replica.cc:333] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:52.692087 21252 raft_consensus.cc:2243] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:52.692739 21252 raft_consensus.cc:2272] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:52.698017 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:52.699852 20275 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 67bcc13a05734fe4abfd011d603d31ad
I20260924 19:58:52.702724 20701 tablet_service.cc:1558] Processing DeleteTablet for tablet 67bcc13a05734fe4abfd011d603d31ad with delete_type TABLET_DATA_DELETED (replaced by tablet 811a1580acf747b49d8db8e2ae38b1f4) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:52.704989 20847 tablet_service.cc:1558] Processing DeleteTablet for tablet 67bcc13a05734fe4abfd011d603d31ad with delete_type TABLET_DATA_DELETED (replaced by tablet 811a1580acf747b49d8db8e2ae38b1f4) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:52.705744 20275 catalog_manager.cc:6934] ReplaceTablet: tablet 67bcc13a05734fe4abfd011d603d31ad deleted and replaced by tablet 811a1580acf747b49d8db8e2ae38b1f4
I20260924 19:58:52.705840 21252 ts_tablet_manager.cc:1916] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:52.563521 20804 leader_election.cc:304] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:52.642953 20868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93f0ea69855145e5bcfd8080d2969dd0" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:52.716643 20868 raft_consensus.cc:2393] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af20c71727a14e398a217d4623d1399a in current term 1: Already voted for candidate 30a4f50fc28c4616ba622ba51b0b2622 in this term.
I20260924 19:58:52.643659 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 8438dd32ff05426e947d9033981a3ea4 with delete_type TABLET_DATA_DELETED (replaced by tablet e6dcbdac79d04942b3ac59dc11a0ed3c) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:52.644449 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58e9ed4d1fee4d3ca2aa1df1e8d645f7" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:52.718816 20867 raft_consensus.cc:3060] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:52.726824 20867 raft_consensus.cc:2468] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:52.728281 20657 leader_election.cc:304] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:52.729578 21145 raft_consensus.cc:2804] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:52.730103 21145 raft_consensus.cc:697] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Becoming Leader. State: Replica: af20c71727a14e398a217d4623d1399a, State: Running, Role: LEADER
I20260924 19:58:52.730953 21145 consensus_queue.cc:237] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.743582 21130 raft_consensus.cc:2804] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:52.744231 21130 raft_consensus.cc:697] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: Becoming Leader. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Running, Role: LEADER
I20260924 19:58:52.745038 21130 consensus_queue.cc:237] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:58:52.750829 20804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.130:38719, user_credentials={real_user=slave}} blocked reactor thread for 187389us
I20260924 19:58:52.755182 20913 heartbeater.cc:344] Connected to a master server at 127.17.2.188:35311
I20260924 19:58:52.755834 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:52.783135 21267 tablet_bootstrap.cc:492] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:52.787959 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet 67bcc13a05734fe4abfd011d603d31ad with delete_type TABLET_DATA_DELETED (replaced by tablet 811a1580acf747b49d8db8e2ae38b1f4) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:52.788810 21255 ts_tablet_manager.cc:1929] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:52.809862 21252 ts_tablet_manager.cc:1929] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:58:52.810426 21255 log.cc:1199] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/b8ec997ee07b4953847c944ee1e9ec64
I20260924 19:58:52.815876 21255 ts_tablet_manager.cc:1950] T b8ec997ee07b4953847c944ee1e9ec64 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:52.824106 21252 log.cc:1199] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/8438dd32ff05426e947d9033981a3ea4
I20260924 19:58:52.814380 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:52.825068 20976 tablet_bootstrap.cc:492] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:52.826078 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58e9ed4d1fee4d3ca2aa1df1e8d645f7" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:52.826642 20574 raft_consensus.cc:3060] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:52.830178 20976 tablet_bootstrap.cc:654] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:52.831652 20658 leader_election.cc:304] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:58:52.833235 21252 ts_tablet_manager.cc:1950] T 8438dd32ff05426e947d9033981a3ea4 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:52.834671 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 8438dd32ff05426e947d9033981a3ea4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:52.835372 20721 raft_consensus.cc:3060] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:52.835676 21252 tablet_replica.cc:333] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:52.836454 21252 raft_consensus.cc:2243] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:52.836898 21252 raft_consensus.cc:2272] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:52.838687 21252 ts_tablet_manager.cc:1916] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:52.839551 21255 tablet_replica.cc:333] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:52.840381 21255 raft_consensus.cc:2243] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:52.841171 21255 pending_rounds.cc:70] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: Trying to abort 1 pending ops.
I20260924 19:58:52.841390 21255 pending_rounds.cc:77] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986604535476224 op_type: NO_OP noop_request { }
I20260924 19:58:52.841756 21255 raft_consensus.cc:2889] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:58:52.842082 21255 raft_consensus.cc:2272] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:52.845432 20278 catalog_manager.cc:2770] deleted HMS catalog entry for table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]
I20260924 19:58:52.851429 21267 tablet_bootstrap.cc:654] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:52.852324 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet b8ec997ee07b4953847c944ee1e9ec64 (table default.table_08eddf6c85b94591a21958b5f18ada13 [id=6539df5762ac4b4f9663be322513b2a0]) successfully deleted
I20260924 19:58:52.854116 20976 tablet_bootstrap.cc:492] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:52.854548 20976 ts_tablet_manager.cc:1403] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.004s
I20260924 19:58:52.856875 20976 raft_consensus.cc:359] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.857451 20976 raft_consensus.cc:385] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:52.857695 20976 raft_consensus.cc:740] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:52.859477 20976 consensus_queue.cc:260] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.863381 21227 tablet_bootstrap.cc:654] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:52.866642 20976 ts_tablet_manager.cc:1434] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20260924 19:58:52.867449 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
W20260924 19:58:52.872424 20277 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b9baeba47f2642b38898c5b2f045cccf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:52.875554 20276 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40798 (request call id 50) took 4193 ms. Trace:
I20260924 19:58:52.875941 20276 rpcz_store.cc:276] 0924 19:58:48.681781 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:48.682069 (+ 288us) service_pool.cc:224] Handling call
0924 19:58:49.868781 (+1186712us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:52.875328 (+3006547us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":899800,"Hive Metastore.run_cpu_time_us":353,"Hive Metastore.run_wall_time_us":386220}
I20260924 19:58:52.876669 21267 tablet_bootstrap.cc:492] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:52.877076 21267 ts_tablet_manager.cc:1403] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.094s user 0.006s sys 0.006s
W20260924 19:58:52.878170 20282 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9067ff1ac3de431993245b7bafa4a8df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:52.882161 20284 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5edadeb38f0143aea238c6775f09b6e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:52.882571 21252 ts_tablet_manager.cc:1929] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:52.882951 21252 log.cc:1199] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/67bcc13a05734fe4abfd011d603d31ad
I20260924 19:58:52.883615 21252 ts_tablet_manager.cc:1950] T 67bcc13a05734fe4abfd011d603d31ad P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:52.884943 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 67bcc13a05734fe4abfd011d603d31ad (table default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]) successfully deleted
I20260924 19:58:52.885568 21267 raft_consensus.cc:359] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.887758 21267 raft_consensus.cc:385] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:58:52.888645 20280 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1c56d0ae4f73481fab5eea13c17efd81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:52.890722 21267 raft_consensus.cc:740] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:52.897735 20438 catalog_manager.cc:2811] Deleting table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29] in response to Hive Metastore notification log event 15
I20260924 19:58:52.892485 21267 consensus_queue.cc:260] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.926018 20574 raft_consensus.cc:2468] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
W20260924 19:58:52.942202 20508 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.131:41601, user_credentials={real_user=slave}} blocked reactor thread for 133652us
I20260924 19:58:52.948704 21201 raft_consensus.cc:2749] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:52.955986 21262 consensus_queue.cc:1048] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:58:52.965155 21227 tablet_bootstrap.cc:492] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:52.965883 21227 ts_tablet_manager.cc:1403] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.282s user 0.010s sys 0.003s
I20260924 19:58:52.968724 21227 raft_consensus.cc:359] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.972259 21227 raft_consensus.cc:385] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:52.972604 21227 raft_consensus.cc:740] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:52.974203 21227 consensus_queue.cc:260] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:52.977612 21227 ts_tablet_manager.cc:1434] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20260924 19:58:52.999012 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c40348984314595b6e6e49c4f03f35e from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:53.000514 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 93f0ea69855145e5bcfd8080d2969dd0 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:53.026206 21193 ts_tablet_manager.cc:1929] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:58:53.040658 21255 ts_tablet_manager.cc:1916] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:53.063290 20702 tablet_service.cc:1511] Processing CreateTablet for tablet e6dcbdac79d04942b3ac59dc11a0ed3c (DEFAULT_TABLE table=default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:53.064308 20702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6dcbdac79d04942b3ac59dc11a0ed3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:53.054198 21159 consensus_queue.cc:1048] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:53.058748 20554 tablet_service.cc:1511] Processing CreateTablet for tablet e6dcbdac79d04942b3ac59dc11a0ed3c (DEFAULT_TABLE table=default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:53.076016 20554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6dcbdac79d04942b3ac59dc11a0ed3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:53.078346 20848 tablet_service.cc:1511] Processing CreateTablet for tablet e6dcbdac79d04942b3ac59dc11a0ed3c (DEFAULT_TABLE table=default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:53.082656 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6dcbdac79d04942b3ac59dc11a0ed3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:53.095501 20276 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 93f0ea69855145e5bcfd8080d2969dd0
I20260924 19:58:53.096899 20276 catalog_manager.cc:6934] ReplaceTablet: tablet 93f0ea69855145e5bcfd8080d2969dd0 deleted and replaced by tablet a623265a78374bb2a3dc20851ae4aaa2
I20260924 19:58:53.100589 20701 tablet_service.cc:1558] Processing DeleteTablet for tablet 93f0ea69855145e5bcfd8080d2969dd0 with delete_type TABLET_DATA_DELETED (replaced by tablet a623265a78374bb2a3dc20851ae4aaa2) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:53.102806 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 93f0ea69855145e5bcfd8080d2969dd0 with delete_type TABLET_DATA_DELETED (replaced by tablet a623265a78374bb2a3dc20851ae4aaa2) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:53.107662 21252 tablet_replica.cc:333] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:53.108486 21252 raft_consensus.cc:2243] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:53.109018 21252 raft_consensus.cc:2272] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:53.111759 21252 ts_tablet_manager.cc:1916] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:53.111534 20283 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_7a523607282c4d1c85c7a42deabb470b" } modify_external_catalogs: true
I20260924 19:58:53.112346 20283 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_7a523607282c4d1c85c7a42deabb470b" } modify_external_catalogs: true
I20260924 19:58:53.116109 20553 tablet_service.cc:1558] Processing DeleteTablet for tablet 93f0ea69855145e5bcfd8080d2969dd0 with delete_type TABLET_DATA_DELETED (replaced by tablet a623265a78374bb2a3dc20851ae4aaa2) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:53.118997 21193 log.cc:1199] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/8438dd32ff05426e947d9033981a3ea4
I20260924 19:58:53.119926 21281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f09f90f3506b48cd9c797dbbeffd0721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:53.120704 21281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:53.121201 21281 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f09f90f3506b48cd9c797dbbeffd0721)
I20260924 19:58:53.121716 21281 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:53.119858 21193 ts_tablet_manager.cc:1950] T 8438dd32ff05426e947d9033981a3ea4 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:53.123401 21244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09f90f3506b48cd9c797dbbeffd0721. Latest consensus state: current_term: 2 leader_uuid: "f09f90f3506b48cd9c797dbbeffd0721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:53.124024 21244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:53.125108 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 8438dd32ff05426e947d9033981a3ea4 (table default.table_92546dd5153b4d58af6fc19d4dee39f0 [id=a5e0426bb5db466a960323a3d0d07a29]) successfully deleted
I20260924 19:58:53.130682 21193 tablet_replica.cc:333] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:53.133057 21193 raft_consensus.cc:2243] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:53.133550 21193 raft_consensus.cc:2272] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:53.158932 21262 raft_consensus.cc:493] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:53.159437 21262 raft_consensus.cc:515] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:53.161412 21262 leader_election.cc:290] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:53.163494 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c54e96fe673f438eb2919be2ad785641" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:53.164211 20722 raft_consensus.cc:2468] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 0.
I20260924 19:58:53.165459 20509 leader_election.cc:304] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:53.166545 21262 raft_consensus.cc:2804] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:53.166919 21262 raft_consensus.cc:493] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:53.167264 21262 raft_consensus.cc:3060] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:53.166769 20275 catalog_manager.cc:5838] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a reported cstate change: term changed from 0 to 1, leader changed from <none> to af20c71727a14e398a217d4623d1399a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "af20c71727a14e398a217d4623d1399a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: UNKNOWN } } }
W20260924 19:58:53.170007 21276 log.cc:927] Time spent T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: Append to log took a long time: real 0.098s user 0.003s sys 0.000s
I20260924 19:58:53.183558 21281 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
I20260924 19:58:53.188139 20438 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:58:53.190824 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet e6dcbdac79d04942b3ac59dc11a0ed3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:52 UTC) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:53.199134 21262 raft_consensus.cc:515] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:53.201630 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:53.207325 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c54e96fe673f438eb2919be2ad785641" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
I20260924 19:58:53.207892 20722 raft_consensus.cc:3060] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:53.210361 20701 tablet_service.cc:1558] Processing DeleteTablet for tablet e6dcbdac79d04942b3ac59dc11a0ed3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:52 UTC) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:53.220250 20553 tablet_service.cc:1558] Processing DeleteTablet for tablet e6dcbdac79d04942b3ac59dc11a0ed3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:52 UTC) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:53.230432 20868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c54e96fe673f438eb2919be2ad785641" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:53.231073 20278 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40798 (request call id 62) took 3185 ms. Trace:
I20260924 19:58:53.231191 20868 raft_consensus.cc:2468] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 0.
I20260924 19:58:53.231467 20278 rpcz_store.cc:276] 0924 19:58:50.045563 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:50.045691 (+ 128us) service_pool.cc:224] Handling call
0924 19:58:52.393830 (+2348139us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:53.231001 (+837171us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1501,"Hive Metastore.run_cpu_time_us":323,"Hive Metastore.run_wall_time_us":448961}
I20260924 19:58:53.237962 21267 ts_tablet_manager.cc:1434] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.357s user 0.026s sys 0.026s
W20260924 19:58:53.259210 20804 consensus_peers.cc:597] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c -> Peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Couldn't send request to peer 30a4f50fc28c4616ba622ba51b0b2622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8438dd32ff05426e947d9033981a3ea4. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:53.263762 21193 ts_tablet_manager.cc:1916] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:53.264446 20275 master_service.cc:960] ReplaceTablet: received request to replace tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:53.268460 21227 tablet_bootstrap.cc:492] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:53.280293 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:53.295730 21267 tablet_bootstrap.cc:492] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:53.266024 20722 raft_consensus.cc:2468] T c54e96fe673f438eb2919be2ad785641 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 1.
I20260924 19:58:53.325477 20509 leader_election.cc:304] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:53.336347 21227 tablet_bootstrap.cc:654] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:53.338107 21159 raft_consensus.cc:2804] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:53.338624 21159 raft_consensus.cc:697] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: Becoming Leader. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Running, Role: LEADER
I20260924 19:58:53.339464 21159 consensus_queue.cc:237] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } }
I20260924 19:58:53.340124 20219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022c1898fcbe4856aa680c26047f6934" candidate_term: 3 candidate_status { last_received { term: 2 index: 89 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:53.266057 21252 ts_tablet_manager.cc:1929] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:53.345173 21252 log.cc:1199] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/93f0ea69855145e5bcfd8080d2969dd0
I20260924 19:58:53.345979 21252 ts_tablet_manager.cc:1950] T 93f0ea69855145e5bcfd8080d2969dd0 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:53.347970 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c54e96fe673f438eb2919be2ad785641" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:53.348544 20867 raft_consensus.cc:3060] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:58:53.355612 20804 consensus_peers.cc:597] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c -> Peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Couldn't send request to peer af20c71727a14e398a217d4623d1399a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8438dd32ff05426e947d9033981a3ea4. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:58:53.356994 21298 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f09f90f3506b48cd9c797dbbeffd0721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:53.357766 21298 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: This master's current role is: LEADER
I20260924 19:58:53.358918 20867 raft_consensus.cc:2468] T c54e96fe673f438eb2919be2ad785641 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 1.
I20260924 19:58:53.374194 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022c1898fcbe4856aa680c26047f6934" candidate_term: 3 candidate_status { last_received { term: 2 index: 89 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:53.376665 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:53.377977 20281 catalog_manager.cc:3541] renamed table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0] in HMS: new name default.table_c8b8f1cd14774827a2cc045e0ffbf35d
I20260924 19:58:53.316411 21262 leader_election.cc:290] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Requested vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:53.385669 21296 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:58:53.398906 21255 ts_tablet_manager.cc:1929] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:53.409719 20271 catalog_manager.cc:5063] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): tablet 93f0ea69855145e5bcfd8080d2969dd0 (table default.table_6ac8d98b233146c495713e0cda0ac6b5 [id=46a1524a039b484d9118dee4b62cb8f9]) successfully deleted
W20260924 19:58:53.411813 20271 catalog_manager.cc:5043] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet e6dcbdac79d04942b3ac59dc11a0ed3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6dcbdac79d04942b3ac59dc11a0ed3c already in progress: creating tablet
I20260924 19:58:53.401599 20976 tablet_bootstrap.cc:492] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:53.428740 20976 tablet_bootstrap.cc:654] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:53.431589 21255 log.cc:1199] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/67bcc13a05734fe4abfd011d603d31ad
I20260924 19:58:53.432417 21255 ts_tablet_manager.cc:1950] T 67bcc13a05734fe4abfd011d603d31ad P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:53.437563 21267 tablet_bootstrap.cc:654] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:53.440150 21193 ts_tablet_manager.cc:1929] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:53.440562 21193 log.cc:1199] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/67bcc13a05734fe4abfd011d603d31ad
I20260924 19:58:53.442703 20275 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7
I20260924 19:58:53.444114 21193 ts_tablet_manager.cc:1950] T 67bcc13a05734fe4abfd011d603d31ad P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:53.445837 20848 tablet_service.cc:1558] Processing DeleteTablet for tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d49e3b1113b4614b359e9922e7fe55f) from {username='slave'} at 127.0.0.1:60248
I20260924 19:58:53.446147 20554 tablet_service.cc:1558] Processing DeleteTablet for tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d49e3b1113b4614b359e9922e7fe55f) from {username='slave'} at 127.0.0.1:48342
I20260924 19:58:53.446776 20270 catalog_manager.cc:5063] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): tablet 67bcc13a05734fe4abfd011d603d31ad (table default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]) successfully deleted
I20260924 19:58:53.449203 20271 catalog_manager.cc:5063] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): tablet 67bcc13a05734fe4abfd011d603d31ad (table default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]) successfully deleted
I20260924 19:58:53.450325 21227 tablet_bootstrap.cc:492] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:53.450764 21227 ts_tablet_manager.cc:1403] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.183s user 0.009s sys 0.004s
I20260924 19:58:53.451659 21193 tablet_replica.cc:333] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:53.452459 21193 raft_consensus.cc:2243] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:53.453465 21193 raft_consensus.cc:2272] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:53.453433 20275 catalog_manager.cc:6934] ReplaceTablet: tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7 deleted and replaced by tablet 0d49e3b1113b4614b359e9922e7fe55f
I20260924 19:58:53.459702 20275 master_service.cc:960] ReplaceTablet: received request to replace tablet 93f0ea69855145e5bcfd8080d2969dd0 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:53.461197 20976 tablet_bootstrap.cc:492] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:53.461350 20554 tablet_service.cc:1511] Processing CreateTablet for tablet a623265a78374bb2a3dc20851ae4aaa2 (DEFAULT_TABLE table=default.table_6ac8d98b233146c495713e0cda0ac6b5 [id=46a1524a039b484d9118dee4b62cb8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:53.461683 20976 ts_tablet_manager.cc:1403] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.004s
I20260924 19:58:53.462378 20554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a623265a78374bb2a3dc20851ae4aaa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:53.465014 20976 raft_consensus.cc:359] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.465770 20976 raft_consensus.cc:385] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:53.466106 20976 raft_consensus.cc:740] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:53.466810 20976 consensus_queue.cc:260] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.468360 21227 raft_consensus.cc:359] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.469049 21227 raft_consensus.cc:385] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:53.469334 21227 raft_consensus.cc:740] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:53.473570 20553 tablet_service.cc:1511] Processing CreateTablet for tablet 811a1580acf747b49d8db8e2ae38b1f4 (DEFAULT_TABLE table=default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:53.474597 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811a1580acf747b49d8db8e2ae38b1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:53.474428 21227 consensus_queue.cc:260] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.483570 21281 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 19:58:53.488796 21176 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" instance_seqno: 1790279919662480) as {username='slave'} at 127.17.2.129:48101; Asking this server to re-register.
I20260924 19:58:53.446271 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d49e3b1113b4614b359e9922e7fe55f) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:53.467468 20701 tablet_service.cc:1511] Processing CreateTablet for tablet a623265a78374bb2a3dc20851ae4aaa2 (DEFAULT_TABLE table=default.table_6ac8d98b233146c495713e0cda0ac6b5 [id=46a1524a039b484d9118dee4b62cb8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:53.490334 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a623265a78374bb2a3dc20851ae4aaa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:53.491063 20620 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:53.477936 20700 tablet_service.cc:1511] Processing CreateTablet for tablet 811a1580acf747b49d8db8e2ae38b1f4 (DEFAULT_TABLE table=default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:53.495968 20700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811a1580acf747b49d8db8e2ae38b1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:53.497511 21175 ts_manager.cc:194] Registered new tserver with Master: 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667)
I20260924 19:58:53.507436 21227 ts_tablet_manager.cc:1434] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.056s user 0.007s sys 0.001s
I20260924 19:58:53.511502 21252 tablet_replica.cc:333] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: stopping tablet replica
I20260924 19:58:53.512763 21252 raft_consensus.cc:2243] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:53.513643 21252 pending_rounds.cc:70] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: Trying to abort 1 pending ops.
I20260924 19:58:53.513967 21252 pending_rounds.cc:77] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986604480512000 op_type: NO_OP noop_request { }
I20260924 19:58:53.514451 21252 raft_consensus.cc:2889] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:58:53.514900 21252 raft_consensus.cc:2272] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:53.514918 20275 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_f25b80aa63f9428d849b3723d9f25a39" } modify_external_catalogs: true
I20260924 19:58:53.515628 20275 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_f25b80aa63f9428d849b3723d9f25a39" } modify_external_catalogs: true
I20260924 19:58:53.518069 21252 ts_tablet_manager.cc:1916] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:53.468871 20976 ts_tablet_manager.cc:1434] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260924 19:58:53.469246 20913 heartbeater.cc:499] Master 127.17.2.188:35311 was elected leader, sending a full tablet report...
I20260924 19:58:53.490254 20848 tablet_service.cc:1511] Processing CreateTablet for tablet a623265a78374bb2a3dc20851ae4aaa2 (DEFAULT_TABLE table=default.table_6ac8d98b233146c495713e0cda0ac6b5 [id=46a1524a039b484d9118dee4b62cb8f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:53.570591 20276 catalog_manager.cc:5838] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129). New cstate: current_term: 1 leader_uuid: "30a4f50fc28c4616ba622ba51b0b2622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: HEALTHY } } }
I20260924 19:58:53.533318 21255 tablet_replica.cc:333] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:53.573235 21255 raft_consensus.cc:2243] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:53.573977 21193 ts_tablet_manager.cc:1916] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:53.574245 21255 raft_consensus.cc:2272] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:53.576730 21255 ts_tablet_manager.cc:1916] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:53.492107 20846 tablet_service.cc:1511] Processing CreateTablet for tablet 811a1580acf747b49d8db8e2ae38b1f4 (DEFAULT_TABLE table=default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]), partition=RANGE (key) PARTITION UNBOUNDED
W20260924 19:58:53.591305 20278 catalog_manager.cc:5671] Ignoring report from unknown tablet e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:58:53.539130 20702 tablet_service.cc:1558] Processing DeleteTablet for tablet 93f0ea69855145e5bcfd8080d2969dd0 with delete_type TABLET_DATA_DELETED (replaced by tablet a623265a78374bb2a3dc20851ae4aaa2) from {username='slave'} at 127.0.0.1:60830
I20260924 19:58:53.597638 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 811a1580acf747b49d8db8e2ae38b1f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:53.545473 21267 tablet_bootstrap.cc:492] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:53.599489 20848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a623265a78374bb2a3dc20851ae4aaa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:53.599491 21267 ts_tablet_manager.cc:1403] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.304s user 0.008s sys 0.003s
I20260924 19:58:53.602327 21267 raft_consensus.cc:359] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.603042 21267 raft_consensus.cc:385] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:53.603349 21267 raft_consensus.cc:740] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:53.607873 21296 catalog_manager.cc:1308] Loaded cluster ID: 543042799ec94022838cc92e16f8796f
I20260924 19:58:53.608132 21296 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: loading cluster ID for follower catalog manager: success
I20260924 19:58:53.610049 21159 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:53.610546 21159 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.612572 21159 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:53.614140 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6dcbdac79d04942b3ac59dc11a0ed3c" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:53.614773 20867 raft_consensus.cc:2468] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 0.
I20260924 19:58:53.616580 20508 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters:
I20260924 19:58:53.616650 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6dcbdac79d04942b3ac59dc11a0ed3c" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:53.618232 21159 raft_consensus.cc:2804] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:53.618650 21159 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:53.619031 21159 raft_consensus.cc:3060] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:53.620237 21267 consensus_queue.cc:260] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.621224 20722 raft_consensus.cc:2468] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 0.
I20260924 19:58:53.630523 21176 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" instance_seqno: 1790279924130258) as {username='slave'} at 127.17.2.131:33145; Asking this server to re-register.
I20260924 19:58:53.635104 21159 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.638157 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6dcbdac79d04942b3ac59dc11a0ed3c" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
I20260924 19:58:53.638475 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e6dcbdac79d04942b3ac59dc11a0ed3c" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:53.638688 20722 raft_consensus.cc:3060] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:53.638995 20867 raft_consensus.cc:3060] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:53.650105 20867 raft_consensus.cc:2468] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 1.
I20260924 19:58:53.650262 20847 tablet_service.cc:1511] Processing CreateTablet for tablet 0d49e3b1113b4614b359e9922e7fe55f (DEFAULT_TABLE table=default.table_7a523607282c4d1c85c7a42deabb470b [id=d9d37737d55c4be39d471fcf624a8866]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:53.651430 20847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d49e3b1113b4614b359e9922e7fe55f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:53.651574 20508 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters:
I20260924 19:58:53.652295 21262 raft_consensus.cc:2804] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:53.652722 21262 raft_consensus.cc:697] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: Becoming Leader. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Running, Role: LEADER
I20260924 19:58:53.653486 21262 consensus_queue.cc:237] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:58:53.660414 20276 catalog_manager.cc:5671] Ignoring report from unknown tablet e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:58:53.668146 20913 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:53.673692 20976 tablet_bootstrap.cc:492] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:53.692189 21227 tablet_bootstrap.cc:492] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:53.681411 20976 tablet_bootstrap.cc:654] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:53.693856 20722 raft_consensus.cc:2468] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a4f50fc28c4616ba622ba51b0b2622 in term 1.
I20260924 19:58:53.703342 21176 ts_manager.cc:194] Registered new tserver with Master: 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:53.710146 21159 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 election: Requested vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:53.714226 21227 tablet_bootstrap.cc:654] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:53.721369 21193 ts_tablet_manager.cc:1929] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:53.721725 21193 log.cc:1199] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/93f0ea69855145e5bcfd8080d2969dd0
I20260924 19:58:53.722414 21193 ts_tablet_manager.cc:1950] T 93f0ea69855145e5bcfd8080d2969dd0 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
W20260924 19:58:53.729270 20276 catalog_manager.cc:5671] Ignoring report from unknown tablet e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:58:53.732892 20278 master_service.cc:960] ReplaceTablet: received request to replace tablet 0c40348984314595b6e6e49c4f03f35e from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:53.734503 20278 master_service.cc:960] ReplaceTablet: received request to replace tablet 635f4961769f49dbb908af7904d29042 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:53.739740 21296 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: acquiring CA information for follower catalog manager: success
I20260924 19:58:53.742893 21267 ts_tablet_manager.cc:1434] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.143s user 0.008s sys 0.000s
I20260924 19:58:53.749752 20976 tablet_bootstrap.cc:492] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:53.750159 20976 ts_tablet_manager.cc:1403] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.004s
I20260924 19:58:53.752586 21296 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:58:53.755846 21227 tablet_bootstrap.cc:492] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:53.756306 21227 ts_tablet_manager.cc:1403] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.009s
W20260924 19:58:53.766247 20276 catalog_manager.cc:5671] Ignoring report from unknown tablet e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:58:53.770392 21176 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "af20c71727a14e398a217d4623d1399a" instance_seqno: 1790279921785028) as {username='slave'} at 127.17.2.130:56691; Asking this server to re-register.
I20260924 19:58:53.772811 20767 heartbeater.cc:461] Registering TS with master...
I20260924 19:58:53.772298 21227 raft_consensus.cc:359] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.773134 21227 raft_consensus.cc:385] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:53.773440 21227 raft_consensus.cc:740] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:53.774163 21227 consensus_queue.cc:260] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.780442 21176 ts_manager.cc:194] Registered new tserver with Master: af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:53.784493 21227 ts_tablet_manager.cc:1434] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260924 19:58:53.795753 21193 tablet_replica.cc:333] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
W20260924 19:58:53.801182 20278 catalog_manager.cc:5671] Ignoring report from unknown tablet e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:58:53.802700 21267 tablet_bootstrap.cc:492] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:53.796707 21227 tablet_bootstrap.cc:492] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:53.805168 21193 raft_consensus.cc:2243] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:58:53.809710 21193 pending_rounds.cc:70] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: Trying to abort 1 pending ops.
I20260924 19:58:53.810065 21193 pending_rounds.cc:77] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332986608255963136 op_type: NO_OP noop_request { }
I20260924 19:58:53.810508 21193 raft_consensus.cc:2889] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260924 19:58:53.810925 21193 raft_consensus.cc:2272] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:53.812060 21227 tablet_bootstrap.cc:654] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:53.827003 21193 ts_tablet_manager.cc:1916] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:53.836647 20976 raft_consensus.cc:359] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.837390 20976 raft_consensus.cc:385] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:53.837639 20976 raft_consensus.cc:740] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:53.838384 20976 consensus_queue.cc:260] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.844983 21227 tablet_bootstrap.cc:492] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:53.845458 21227 ts_tablet_manager.cc:1403] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.060s user 0.007s sys 0.004s
I20260924 19:58:53.848322 21227 raft_consensus.cc:359] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.848994 21227 raft_consensus.cc:385] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:53.849274 21227 raft_consensus.cc:740] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
I20260924 19:58:53.849997 21227 consensus_queue.cc:260] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.854671 21267 tablet_bootstrap.cc:654] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:53.864907 21227 ts_tablet_manager.cc:1434] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.019s user 0.004s sys 0.004s
I20260924 19:58:53.867468 21255 ts_tablet_manager.cc:1929] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:58:53.868206 21255 log.cc:1199] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/8438dd32ff05426e947d9033981a3ea4
W20260924 19:58:53.874154 20270 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 5fa9ceda714341e5b8eeb4144fea1342 (table default.table_1ed870b0a11e446c86e6a4c6a15e81b6 [id=966de0fe489d49ea9ae3954d0d0916c0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5fa9ceda714341e5b8eeb4144fea1342 consensus state has no leader
I20260924 19:58:53.882335 20976 ts_tablet_manager.cc:1434] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.132s user 0.003s sys 0.004s
I20260924 19:58:53.906339 20976 tablet_bootstrap.cc:492] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:53.890214 21252 ts_tablet_manager.cc:1929] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:53.919255 21252 log.cc:1199] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-1/wal/wals/58e9ed4d1fee4d3ca2aa1df1e8d645f7
I20260924 19:58:53.920291 21252 ts_tablet_manager.cc:1950] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P af20c71727a14e398a217d4623d1399a: Deleting consensus metadata
I20260924 19:58:53.924088 21255 ts_tablet_manager.cc:1950] T 8438dd32ff05426e947d9033981a3ea4 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:53.926461 21255 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60248 (request call id 68) took 1283 ms. Trace:
I20260924 19:58:53.945771 21262 raft_consensus.cc:493] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:53.946220 21262 raft_consensus.cc:515] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:53.947523 20278 master_service.cc:960] ReplaceTablet: received request to replace tablet 65e0685bc4564703b88a5247511686ec from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:53.948169 21262 leader_election.cc:290] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:53.951016 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a623265a78374bb2a3dc20851ae4aaa2" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:53.951787 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a623265a78374bb2a3dc20851ae4aaa2" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
W20260924 19:58:53.952661 20509 leader_election.cc:343] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:58:53.953169 20508 leader_election.cc:343] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:53.953511 20508 leader_election.cc:304] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:58:53.954982 21262 raft_consensus.cc:2749] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:53.955024 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 635f4961769f49dbb908af7904d29042 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:53.927034 21255 rpcz_store.cc:276] 0924 19:58:52.643246 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:52.643394 (+ 148us) service_pool.cc:224] Handling call
0924 19:58:53.445591 (+802197us) ts_tablet_manager.cc:1151] Deleting tablet 8438dd32ff05426e947d9033981a3ea4
0924 19:58:53.523202 (+ 77611us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:58:53.689598 (+166396us) tablet_metadata.cc:665] Metadata flushed
0924 19:58:53.867252 (+177654us) tablet_metadata.cc:665] Metadata flushed
0924 19:58:53.926388 (+ 59136us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":727046}
I20260924 19:58:53.980432 20976 tablet_bootstrap.cc:654] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:53.981681 20278 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 65e0685bc4564703b88a5247511686ec
I20260924 19:58:53.985612 20278 catalog_manager.cc:6934] ReplaceTablet: tablet 65e0685bc4564703b88a5247511686ec deleted and replaced by tablet ac71ad6f430d4bfab1cdb6ec6163e656
I20260924 19:58:53.987846 21267 tablet_bootstrap.cc:492] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:53.988905 21267 ts_tablet_manager.cc:1403] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.187s user 0.013s sys 0.000s
I20260924 19:58:53.978345 21255 tablet_replica.cc:333] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:53.999271 21255 raft_consensus.cc:2243] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:54.006465 21255 raft_consensus.cc:2272] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:54.043329 21267 raft_consensus.cc:359] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.047966 21255 ts_tablet_manager.cc:1916] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:54.044015 21267 raft_consensus.cc:385] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:54.058976 20279 catalog_manager.cc:3541] renamed table default.table_14cbcbeb7f98417197bb13b1f8b5af49 [id=c2d09462b2b14d1a862ef2b18152c4e1] in HMS: new name default.table_77566e1a46ec4a4ba7dde05d53b7c5cc
I20260924 19:58:54.055634 21267 raft_consensus.cc:740] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:54.062404 20976 tablet_bootstrap.cc:492] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:54.062830 20976 ts_tablet_manager.cc:1403] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.157s user 0.007s sys 0.004s
I20260924 19:58:54.065176 20976 raft_consensus.cc:359] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.065775 20976 raft_consensus.cc:385] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:54.066030 20976 raft_consensus.cc:740] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:54.070322 21267 consensus_queue.cc:260] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.101377 21193 ts_tablet_manager.cc:1929] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:54.101815 21193 log.cc:1199] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:58:54.102573 21193 ts_tablet_manager.cc:1950] T e6dcbdac79d04942b3ac59dc11a0ed3c P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:54.108131 21267 ts_tablet_manager.cc:1434] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.117s user 0.007s sys 0.002s
I20260924 19:58:54.113847 21262 raft_consensus.cc:493] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:54.114362 21262 raft_consensus.cc:515] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.116451 21262 leader_election.cc:290] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:54.119537 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811a1580acf747b49d8db8e2ae38b1f4" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:54.119940 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811a1580acf747b49d8db8e2ae38b1f4" candidate_uuid: "30a4f50fc28c4616ba622ba51b0b2622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
W20260924 19:58:54.121075 20508 leader_election.cc:343] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Illegal state: must be running to vote when last-logged opid is not known
W20260924 19:58:54.121474 20509 leader_election.cc:343] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:58:54.121860 20509 leader_election.cc:304] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:58:54.123739 21262 raft_consensus.cc:2749] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:54.066653 20976 consensus_queue.cc:260] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.131691 20976 ts_tablet_manager.cc:1434] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.069s user 0.007s sys 0.000s
I20260924 19:58:54.133409 20976 tablet_bootstrap.cc:492] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:54.144191 21267 tablet_bootstrap.cc:492] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:54.169600 20278 master_service.cc:960] ReplaceTablet: received request to replace tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:54.186616 21193 tablet_replica.cc:333] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622: stopping tablet replica
I20260924 19:58:54.197690 20278 master_service.cc:960] ReplaceTablet: received request to replace tablet 811a1580acf747b49d8db8e2ae38b1f4 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:54.222298 20278 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 811a1580acf747b49d8db8e2ae38b1f4
I20260924 19:58:54.223805 20278 catalog_manager.cc:6934] ReplaceTablet: tablet 811a1580acf747b49d8db8e2ae38b1f4 deleted and replaced by tablet 5fa3110891954347a1cff0c06a23de63
I20260924 19:58:54.230733 21193 raft_consensus.cc:2243] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:54.231980 21193 raft_consensus.cc:2272] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:54.236234 20282 catalog_manager.cc:2213] created HMS catalog entry for table [id=c2d80fec62674432b58189903dc8e9d2]
I20260924 19:58:54.240708 21193 ts_tablet_manager.cc:1916] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:54.244908 21130 raft_consensus.cc:493] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:54.245199 20976 tablet_bootstrap.cc:654] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:54.245374 21130 raft_consensus.cc:515] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.247531 21130 leader_election.cc:290] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:54.249440 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a623265a78374bb2a3dc20851ae4aaa2" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:54.250084 20574 raft_consensus.cc:2468] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
I20260924 19:58:54.251210 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a623265a78374bb2a3dc20851ae4aaa2" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a" is_pre_election: true
I20260924 19:58:54.251453 20804 leader_election.cc:304] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters:
I20260924 19:58:54.251848 20722 raft_consensus.cc:2468] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 0.
I20260924 19:58:54.252298 21130 raft_consensus.cc:2804] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:54.252588 21130 raft_consensus.cc:493] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:54.252969 21130 raft_consensus.cc:3060] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:54.260634 20282 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 75) took 2150 ms. Trace:
I20260924 19:58:54.248133 21267 tablet_bootstrap.cc:654] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:54.261004 20282 rpcz_store.cc:276] 0924 19:58:52.110697 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:52.110843 (+ 146us) service_pool.cc:224] Handling call
0924 19:58:52.878634 (+767791us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:52.880416 (+ 1782us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:54.236194 (+1355778us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:54.259867 (+ 23673us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:58:54.260247 (+ 380us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:58:54.260564 (+ 317us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:54.238001 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:54.238397 (+ 396us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:54.238420 (+ 23us) write_op.cc:435] Acquired schema lock
0924 19:58:54.238433 (+ 13us) tablet.cc:662] Decoding operations
0924 19:58:54.238689 (+ 256us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:54.238726 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:54.238746 (+ 20us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:54.238963 (+ 217us) tablet.cc:701] Row locks acquired
0924 19:58:54.238977 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:58:54.239091 (+ 114us) write_op.cc:270] Start()
0924 19:58:54.239196 (+ 105us) write_op.cc:276] Timestamp: P: 1790279934238889 usec, L: 0
0924 19:58:54.239210 (+ 14us) op_driver.cc:348] REPLICATION: starting
0924 19:58:54.239673 (+ 463us) log.cc:844] Serialized 432 byte log entry
0924 19:58:54.256949 (+ 17276us) op_driver.cc:464] REPLICATION: finished
0924 19:58:54.257330 (+ 381us) write_op.cc:301] APPLY: starting
0924 19:58:54.257385 (+ 55us) tablet.cc:1394] starting BulkCheckPresence
0924 19:58:54.257485 (+ 100us) tablet.cc:1397] finished BulkCheckPresence
0924 19:58:54.257573 (+ 88us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:58:54.257942 (+ 369us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:58:54.258087 (+ 145us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:58:54.258128 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:58:54.258305 (+ 177us) log.cc:844] Serialized 28 byte log entry
0924 19:58:54.258454 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:54.258522 (+ 68us) write_op.cc:454] Released schema lock
0924 19:58:54.258552 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0924 19:58:54.258761 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:54.258784 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1177169,"Hive Metastore.run_cpu_time_us":635,"Hive Metastore.run_wall_time_us":177244,"child_traces":[["op",{"apply.queue_time_us":230,"apply.run_cpu_time_us":1397,"apply.run_wall_time_us":1397,"num_ops":2,"prepare.queue_time_us":183,"prepare.run_cpu_time_us":1953,"prepare.run_wall_time_us":1953,"raft.queue_time_us":998,"raft.run_cpu_time_us":688,"raft.run_wall_time_us":688,"replication_time_us":17646}]]}
I20260924 19:58:54.275370 21130 raft_consensus.cc:515] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.279821 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a623265a78374bb2a3dc20851ae4aaa2" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:54.280424 20574 raft_consensus.cc:3060] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:54.281538 20722 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a623265a78374bb2a3dc20851ae4aaa2" candidate_uuid: "0055218dbcbd48b58018e9774e59e14c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af20c71727a14e398a217d4623d1399a"
I20260924 19:58:54.282053 20722 raft_consensus.cc:3060] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:54.288709 20574 raft_consensus.cc:2468] T a623265a78374bb2a3dc20851ae4aaa2 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 1.
I20260924 19:58:54.290359 20804 leader_election.cc:304] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622; no voters:
I20260924 19:58:54.291072 20722 raft_consensus.cc:2468] T a623265a78374bb2a3dc20851ae4aaa2 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0055218dbcbd48b58018e9774e59e14c in term 1.
I20260924 19:58:54.335501 21130 leader_election.cc:290] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:54.336123 21130 raft_consensus.cc:2804] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:54.336544 21130 raft_consensus.cc:697] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [term 1 LEADER]: Becoming Leader. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Running, Role: LEADER
I20260924 19:58:54.337313 21130 consensus_queue.cc:237] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.341252 21267 tablet_bootstrap.cc:492] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:54.341674 21267 ts_tablet_manager.cc:1403] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.198s user 0.007s sys 0.004s
I20260924 19:58:54.343879 21255 ts_tablet_manager.cc:1929] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260924 19:58:54.344251 21255 log.cc:1199] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/93f0ea69855145e5bcfd8080d2969dd0
I20260924 19:58:54.344970 21255 ts_tablet_manager.cc:1950] T 93f0ea69855145e5bcfd8080d2969dd0 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
W20260924 19:58:54.347664 20804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.129:33667, user_credentials={real_user=slave}} blocked reactor thread for 57421.2us
I20260924 19:58:54.362473 20976 tablet_bootstrap.cc:492] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:54.362908 20976 ts_tablet_manager.cc:1403] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.230s user 0.011s sys 0.000s
I20260924 19:58:54.365514 20976 raft_consensus.cc:359] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.371770 20976 raft_consensus.cc:385] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:54.372036 20976 raft_consensus.cc:740] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:54.372730 20976 consensus_queue.cc:260] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.374622 20976 ts_tablet_manager.cc:1434] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260924 19:58:54.380508 20282 master_service.cc:960] ReplaceTablet: received request to replace tablet 65e0685bc4564703b88a5247511686ec from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:54.383728 21267 raft_consensus.cc:359] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.384428 21267 raft_consensus.cc:385] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:54.384697 21267 raft_consensus.cc:740] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:54.385351 21267 consensus_queue.cc:260] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.346926 21255 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60248 (request call id 71) took 1244 ms. Trace:
I20260924 19:58:54.395279 21255 rpcz_store.cc:276] 0924 19:58:53.102523 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:53.102667 (+ 144us) service_pool.cc:224] Handling call
0924 19:58:53.977941 (+875274us) ts_tablet_manager.cc:1151] Deleting tablet 93f0ea69855145e5bcfd8080d2969dd0
0924 19:58:53.978061 (+ 120us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:58:54.207528 (+229467us) tablet_metadata.cc:665] Metadata flushed
0924 19:58:54.343741 (+136213us) tablet_metadata.cc:665] Metadata flushed
0924 19:58:54.346849 (+ 3108us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":874610}
I20260924 19:58:54.396528 21255 tablet_replica.cc:333] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:54.397331 21255 raft_consensus.cc:2243] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:54.397763 20284 catalog_manager.cc:2213] created HMS catalog entry for table [id=3395dfcca2e84761aa9f9ec23fbd10cf]
I20260924 19:58:54.397797 21255 raft_consensus.cc:2272] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:58:54.410864 20284 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 78) took 2031 ms. Trace:
I20260924 19:58:54.411238 20284 rpcz_store.cc:276] 0924 19:58:52.379552 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:52.380075 (+ 523us) service_pool.cc:224] Handling call
0924 19:58:52.882607 (+502532us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:52.887248 (+ 4641us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:54.397717 (+1510469us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:54.410187 (+ 12470us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:58:54.410511 (+ 324us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:58:54.410799 (+ 288us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:54.399534 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:54.399924 (+ 390us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:54.399946 (+ 22us) write_op.cc:435] Acquired schema lock
0924 19:58:54.399958 (+ 12us) tablet.cc:662] Decoding operations
0924 19:58:54.400201 (+ 243us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:54.400238 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:54.400259 (+ 21us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:54.400432 (+ 173us) tablet.cc:701] Row locks acquired
0924 19:58:54.400445 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:58:54.400549 (+ 104us) write_op.cc:270] Start()
0924 19:58:54.400625 (+ 76us) write_op.cc:276] Timestamp: P: 1790279934400364 usec, L: 0
0924 19:58:54.400639 (+ 14us) op_driver.cc:348] REPLICATION: starting
0924 19:58:54.401093 (+ 454us) log.cc:844] Serialized 432 byte log entry
0924 19:58:54.408299 (+ 7206us) op_driver.cc:464] REPLICATION: finished
0924 19:58:54.408594 (+ 295us) write_op.cc:301] APPLY: starting
0924 19:58:54.408649 (+ 55us) tablet.cc:1394] starting BulkCheckPresence
0924 19:58:54.408750 (+ 101us) tablet.cc:1397] finished BulkCheckPresence
0924 19:58:54.408838 (+ 88us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:58:54.409239 (+ 401us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:58:54.409370 (+ 131us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:58:54.409411 (+ 41us) write_op.cc:312] APPLY: finished
0924 19:58:54.409584 (+ 173us) log.cc:844] Serialized 28 byte log entry
0924 19:58:54.409718 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:54.409784 (+ 66us) write_op.cc:454] Released schema lock
0924 19:58:54.409813 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0924 19:58:54.410028 (+ 215us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:54.410051 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1347846,"Hive Metastore.run_cpu_time_us":646,"Hive Metastore.run_wall_time_us":161985,"child_traces":[["op",{"apply.queue_time_us":167,"apply.run_cpu_time_us":1399,"apply.run_wall_time_us":1398,"num_ops":2,"prepare.queue_time_us":120,"prepare.run_cpu_time_us":1818,"prepare.run_wall_time_us":1819,"raft.queue_time_us":162,"raft.run_cpu_time_us":603,"raft.run_wall_time_us":602,"replication_time_us":7576}]]}
I20260924 19:58:54.437659 21262 consensus_queue.cc:1048] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:54.438889 20282 master_service.cc:960] ReplaceTablet: received request to replace tablet 65e0685bc4564703b88a5247511686ec from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:54.464730 21255 ts_tablet_manager.cc:1916] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:54.468406 21267 ts_tablet_manager.cc:1434] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.126s user 0.003s sys 0.006s
I20260924 19:58:54.504441 21318 consensus_queue.cc:1048] T c54e96fe673f438eb2919be2ad785641 P 30a4f50fc28c4616ba622ba51b0b2622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:54.508033 20284 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_b9502ead412146d497587b0c995526fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:54.555932 21321 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f09f90f3506b48cd9c797dbbeffd0721)
I20260924 19:58:54.556694 21321 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:54.560307 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022c1898fcbe4856aa680c26047f6934" candidate_term: 3 candidate_status { last_received { term: 2 index: 106 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:54.562832 21321 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
I20260924 19:58:54.577359 21193 ts_tablet_manager.cc:1929] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:54.577932 21193 log.cc:1199] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-0/wal/wals/58e9ed4d1fee4d3ca2aa1df1e8d645f7
I20260924 19:58:54.578764 21193 ts_tablet_manager.cc:1950] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 30a4f50fc28c4616ba622ba51b0b2622: Deleting consensus metadata
I20260924 19:58:54.581064 21193 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48342 (request call id 75) took 1135 ms. Trace:
I20260924 19:58:54.592298 20282 master_service.cc:960] ReplaceTablet: received request to replace tablet a623265a78374bb2a3dc20851ae4aaa2 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:54.602397 20280 catalog_manager.cc:2213] created HMS catalog entry for table [id=3914fcd32f4e4eea934bfb6b273de449]
I20260924 19:58:54.618928 20280 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 69) took 3825 ms. Trace:
I20260924 19:58:54.606063 21193 rpcz_store.cc:276] 0924 19:58:53.445915 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:53.446016 (+ 101us) service_pool.cc:224] Handling call
0924 19:58:54.186447 (+740431us) ts_tablet_manager.cc:1151] Deleting tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7
0924 19:58:54.186498 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:58:54.457774 (+271276us) tablet_metadata.cc:665] Metadata flushed
0924 19:58:54.577197 (+119423us) tablet_metadata.cc:665] Metadata flushed
0924 19:58:54.580990 (+ 3793us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":739899}
I20260924 19:58:54.619735 20278 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_d7b4827d2d284d37b72f600e797d818d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:54.619305 20280 rpcz_store.cc:276] 0924 19:58:50.793256 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:50.824596 (+ 31340us) service_pool.cc:224] Handling call
0924 19:58:52.889101 (+2064505us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:52.891001 (+ 1900us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:54.602351 (+1711350us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:54.618194 (+ 15843us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:58:54.618515 (+ 321us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:58:54.618846 (+ 331us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:54.604500 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:54.604876 (+ 376us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:54.604897 (+ 21us) write_op.cc:435] Acquired schema lock
0924 19:58:54.604910 (+ 13us) tablet.cc:662] Decoding operations
0924 19:58:54.605161 (+ 251us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:54.605199 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:54.605222 (+ 23us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:54.605416 (+ 194us) tablet.cc:701] Row locks acquired
0924 19:58:54.605430 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:58:54.605550 (+ 120us) write_op.cc:270] Start()
0924 19:58:54.605629 (+ 79us) write_op.cc:276] Timestamp: P: 1790279934605349 usec, L: 0
0924 19:58:54.605645 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:58:54.606124 (+ 479us) log.cc:844] Serialized 432 byte log entry
0924 19:58:54.614466 (+ 8342us) op_driver.cc:464] REPLICATION: finished
0924 19:58:54.614801 (+ 335us) write_op.cc:301] APPLY: starting
0924 19:58:54.614859 (+ 58us) tablet.cc:1394] starting BulkCheckPresence
0924 19:58:54.614979 (+ 120us) tablet.cc:1397] finished BulkCheckPresence
0924 19:58:54.615092 (+ 113us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:58:54.615517 (+ 425us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:58:54.615657 (+ 140us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:58:54.615704 (+ 47us) write_op.cc:312] APPLY: finished
0924 19:58:54.615882 (+ 178us) log.cc:844] Serialized 28 byte log entry
0924 19:58:54.616050 (+ 168us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:54.616122 (+ 72us) write_op.cc:454] Released schema lock
0924 19:58:54.616175 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0924 19:58:54.616402 (+ 227us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:54.616427 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1501694,"Hive Metastore.run_cpu_time_us":625,"Hive Metastore.run_wall_time_us":204242,"child_traces":[["op",{"apply.queue_time_us":193,"apply.run_cpu_time_us":1551,"apply.run_wall_time_us":1567,"num_ops":2,"prepare.queue_time_us":179,"prepare.run_cpu_time_us":1910,"prepare.run_wall_time_us":1911,"raft.queue_time_us":1183,"raft.run_cpu_time_us":603,"raft.run_wall_time_us":603,"replication_time_us":8735}]]}
I20260924 19:58:54.656137 20280 master_service.cc:960] ReplaceTablet: received request to replace tablet 635f4961769f49dbb908af7904d29042 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:54.660991 21255 ts_tablet_manager.cc:1929] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:54.661902 21255 log.cc:1199] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:58:54.663594 21255 ts_tablet_manager.cc:1950] T e6dcbdac79d04942b3ac59dc11a0ed3c P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:54.668982 21255 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60248 (request call id 72) took 1478 ms. Trace:
I20260924 19:58:54.669792 21255 rpcz_store.cc:276] 0924 19:58:53.190530 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:53.190674 (+ 144us) service_pool.cc:224] Handling call
0924 19:58:54.396367 (+1205693us) ts_tablet_manager.cc:1151] Deleting tablet e6dcbdac79d04942b3ac59dc11a0ed3c
0924 19:58:54.396416 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:58:54.598631 (+202215us) tablet_metadata.cc:665] Metadata flushed
0924 19:58:54.660686 (+ 62055us) tablet_metadata.cc:665] Metadata flushed
0924 19:58:54.668797 (+ 8111us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205100}
I20260924 19:58:54.672963 21255 tablet_replica.cc:333] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c: stopping tablet replica
I20260924 19:58:54.674213 20282 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet a623265a78374bb2a3dc20851ae4aaa2
I20260924 19:58:54.676693 21130 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:54.677131 21130 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.678997 21130 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:58:54.679592 20282 catalog_manager.cc:6934] ReplaceTablet: tablet a623265a78374bb2a3dc20851ae4aaa2 deleted and replaced by tablet f050831a13f24241a9f314545d1f5457
I20260924 19:58:54.679602 21255 raft_consensus.cc:2243] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:58:54.681459 21255 raft_consensus.cc:2272] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:58:54.684412 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 19:58:54.685060 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 19:58:54.685415 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:58:54.685751 21255 ts_tablet_manager.cc:1916] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:58:54.687525 21130 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:54.796926 21255 ts_tablet_manager.cc:1929] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260924 19:58:54.800269 21255 log.cc:1199] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c: Deleting WAL directory at /tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/ts-2/wal/wals/58e9ed4d1fee4d3ca2aa1df1e8d645f7
I20260924 19:58:54.806249 21255 ts_tablet_manager.cc:1950] T 58e9ed4d1fee4d3ca2aa1df1e8d645f7 P 0055218dbcbd48b58018e9774e59e14c: Deleting consensus metadata
I20260924 19:58:54.824249 21255 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60248 (request call id 73) took 1378 ms. Trace:
I20260924 19:58:54.838680 20282 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_9a44188ed4234e4e98ebdcb2e803b418"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:54.826289 21255 rpcz_store.cc:276] 0924 19:58:53.445584 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:53.445690 (+ 106us) service_pool.cc:224] Handling call
0924 19:58:54.672586 (+1226896us) ts_tablet_manager.cc:1151] Deleting tablet 58e9ed4d1fee4d3ca2aa1df1e8d645f7
0924 19:58:54.672677 (+ 91us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0924 19:58:54.730847 (+ 58170us) tablet_metadata.cc:665] Metadata flushed
0924 19:58:54.795864 (+ 65017us) tablet_metadata.cc:665] Metadata flushed
0924 19:58:54.823850 (+ 27986us) inbound_call.cc:177] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1226184}
I20260924 19:58:54.845386 20277 catalog_manager.cc:2213] created HMS catalog entry for table [id=c76d82558bd147a6b85aed1a31ef4f49]
I20260924 19:58:54.875694 20280 master_service.cc:960] ReplaceTablet: received request to replace tablet e2719ada60a143c6862e90d81f9c2d82 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:54.885078 20277 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 71) took 3839 ms. Trace:
I20260924 19:58:54.889199 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet a623265a78374bb2a3dc20851ae4aaa2 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:54.885425 20277 rpcz_store.cc:276] 0924 19:58:51.045775 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:51.049630 (+ 3855us) service_pool.cc:224] Handling call
0924 19:58:52.872956 (+1823326us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:52.874796 (+ 1840us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:54.845344 (+1970548us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:54.877518 (+ 32174us) catalog_manager.cc:2246] Wrote table and tablets to system table
0924 19:58:54.884687 (+ 7169us) catalog_manager.cc:2280] Inserted table and tablets into CatalogManager maps
0924 19:58:54.885018 (+ 331us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:54.847263 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:54.847651 (+ 388us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:54.847676 (+ 25us) write_op.cc:435] Acquired schema lock
0924 19:58:54.847688 (+ 12us) tablet.cc:662] Decoding operations
0924 19:58:54.847956 (+ 268us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:54.847992 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:54.848017 (+ 25us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:54.848202 (+ 185us) tablet.cc:701] Row locks acquired
0924 19:58:54.848216 (+ 14us) write_op.cc:260] PREPARE: finished
0924 19:58:54.848328 (+ 112us) write_op.cc:270] Start()
0924 19:58:54.848408 (+ 80us) write_op.cc:276] Timestamp: P: 1790279934848137 usec, L: 0
0924 19:58:54.848423 (+ 15us) op_driver.cc:348] REPLICATION: starting
0924 19:58:54.848924 (+ 501us) log.cc:844] Serialized 432 byte log entry
0924 19:58:54.872080 (+ 23156us) op_driver.cc:464] REPLICATION: finished
0924 19:58:54.874063 (+ 1983us) write_op.cc:301] APPLY: starting
0924 19:58:54.874123 (+ 60us) tablet.cc:1394] starting BulkCheckPresence
0924 19:58:54.874229 (+ 106us) tablet.cc:1397] finished BulkCheckPresence
0924 19:58:54.874339 (+ 110us) tablet.cc:1399] starting ApplyRowOperation cycle
0924 19:58:54.874720 (+ 381us) tablet.cc:1410] finished ApplyRowOperation cycle
0924 19:58:54.874858 (+ 138us) tablet_metrics.cc:600] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0924 19:58:54.874910 (+ 52us) write_op.cc:312] APPLY: finished
0924 19:58:54.875102 (+ 192us) log.cc:844] Serialized 28 byte log entry
0924 19:58:54.875261 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:54.875335 (+ 74us) write_op.cc:454] Released schema lock
0924 19:58:54.875366 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0924 19:58:54.875595 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:54.875625 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1702427,"Hive Metastore.run_cpu_time_us":719,"Hive Metastore.run_wall_time_us":243139,"child_traces":[["op",{"apply.queue_time_us":1821,"apply.run_cpu_time_us":1496,"apply.run_wall_time_us":1497,"num_ops":2,"prepare.queue_time_us":118,"prepare.run_cpu_time_us":1939,"prepare.run_wall_time_us":1939,"raft.queue_time_us":1199,"raft.run_cpu_time_us":723,"raft.run_wall_time_us":722,"replication_time_us":23559}]]}
I20260924 19:58:54.913523 20280 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet e2719ada60a143c6862e90d81f9c2d82
I20260924 19:58:54.925189 20280 catalog_manager.cc:6934] ReplaceTablet: tablet e2719ada60a143c6862e90d81f9c2d82 deleted and replaced by tablet 4477d9f1fa554d4b8ae762bfdcfedd44
W20260924 19:58:54.941733 20438 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac71ad6f430d4bfab1cdb6ec6163e656 consensus state has no leader
I20260924 19:58:54.966773 21308 raft_consensus.cc:493] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:54.967293 21308 raft_consensus.cc:515] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:54.969292 21308 leader_election.cc:290] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:54.972105 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811a1580acf747b49d8db8e2ae38b1f4" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:54.972683 20867 raft_consensus.cc:2468] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:54.972633 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811a1580acf747b49d8db8e2ae38b1f4" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:54.973222 20574 raft_consensus.cc:2468] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:54.973933 20657 leader_election.cc:304] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:54.975001 21308 raft_consensus.cc:2804] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:54.975406 21308 raft_consensus.cc:493] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:54.975796 21308 raft_consensus.cc:3060] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:58:54.988937 20438 catalog_manager.cc:6106] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 has no consensus state
I20260924 19:58:55.018698 21262 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:55.019186 21262 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:55.021122 21262 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:58:55.023377 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 19:58:55.028182 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:58:55.028558 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:58:55.029265 21262 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:55.023509 21308 raft_consensus.cc:515] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:58:55.046878 20268 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 consensus state has no leader
I20260924 19:58:55.049839 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811a1580acf747b49d8db8e2ae38b1f4" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:55.050426 20574 raft_consensus.cc:3060] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:55.052994 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "811a1580acf747b49d8db8e2ae38b1f4" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:55.053501 20867 raft_consensus.cc:3060] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:55.059710 21308 leader_election.cc:290] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:55.057312 20574 raft_consensus.cc:2468] T 811a1580acf747b49d8db8e2ae38b1f4 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:55.063313 20658 leader_election.cc:304] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:55.063475 20867 raft_consensus.cc:2468] T 811a1580acf747b49d8db8e2ae38b1f4 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:55.064365 21308 raft_consensus.cc:2804] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:55.091831 21308 raft_consensus.cc:697] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Becoming Leader. State: Replica: af20c71727a14e398a217d4623d1399a, State: Running, Role: LEADER
I20260924 19:58:55.093314 21308 consensus_queue.cc:237] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:55.110666 20277 master_service.cc:960] ReplaceTablet: received request to replace tablet 635f4961769f49dbb908af7904d29042 from {username='slave'} at 127.0.0.1:40798
W20260924 19:58:55.143579 21148 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 0.574s (SENT)
I20260924 19:58:55.146557 20277 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40798:
name: "default.Table_d876eec98f7a4157b29a3241d303ab96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:55.149559 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet c54e96fe673f438eb2919be2ad785641 from {username='slave'} at 127.0.0.1:40798
W20260924 19:58:55.170379 20280 catalog_manager.cc:5671] Ignoring report from unknown tablet 811a1580acf747b49d8db8e2ae38b1f4
I20260924 19:58:55.180990 20276 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet c54e96fe673f438eb2919be2ad785641
I20260924 19:58:55.186510 20276 catalog_manager.cc:6934] ReplaceTablet: tablet c54e96fe673f438eb2919be2ad785641 deleted and replaced by tablet 7a22f69f014340b0a9d775c4d6688e21
I20260924 19:58:55.219599 20283 catalog_manager.cc:2770] deleted HMS catalog entry for table default.table_7a523607282c4d1c85c7a42deabb470b [id=d9d37737d55c4be39d471fcf624a8866]
W20260924 19:58:55.220790 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 115ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:58:55.226475 20279 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40798 (request call id 65) took 5078 ms. Trace:
W20260924 19:58:55.228022 20284 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b9502ead412146d497587b0c995526fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:55.232257 20278 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d7b4827d2d284d37b72f600e797d818d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:55.233621 20282 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9a44188ed4234e4e98ebdcb2e803b418 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:55.237416 21130 consensus_queue.cc:1048] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:55.226764 20279 rpcz_store.cc:276] 0924 19:58:50.148014 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:50.148157 (+ 143us) service_pool.cc:224] Handling call
0924 19:58:52.869077 (+2720920us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:55.224105 (+2355028us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:55.225606 (+ 1501us) catalog_manager.cc:3764] Apply alter schema
0924 19:58:55.226374 (+ 768us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":507643,"Hive Metastore.run_cpu_time_us":1850,"Hive Metastore.run_wall_time_us":680495}
I20260924 19:58:55.261247 20281 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40798 (request call id 63) took 5208 ms. Trace:
I20260924 19:58:55.261610 20281 rpcz_store.cc:276] 0924 19:58:50.052559 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:50.057127 (+ 4568us) service_pool.cc:224] Handling call
0924 19:58:52.867621 (+2810494us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:55.234167 (+2366546us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:55.259979 (+ 25812us) catalog_manager.cc:3764] Apply alter schema
0924 19:58:55.261173 (+ 1194us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":22898,"Hive Metastore.run_cpu_time_us":2268,"Hive Metastore.run_wall_time_us":485797}
I20260924 19:58:55.279099 20438 catalog_manager.cc:2811] Deleting table default.table_7a523607282c4d1c85c7a42deabb470b [id=d9d37737d55c4be39d471fcf624a8866] in response to Hive Metastore notification log event 22
W20260924 19:58:55.293188 20281 catalog_manager.cc:5671] Ignoring report from unknown tablet a623265a78374bb2a3dc20851ae4aaa2
I20260924 19:58:55.324088 20281 master_service.cc:960] ReplaceTablet: received request to replace tablet 635f4961769f49dbb908af7904d29042 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:55.361387 20438 catalog_manager.cc:6133] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Sending DeleteTablet for 3 replicas of tablet 0d49e3b1113b4614b359e9922e7fe55f
W20260924 19:58:55.363888 20277 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d876eec98f7a4157b29a3241d303ab96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:55.366034 20283 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40798 (request call id 83) took 2278 ms. Trace:
I20260924 19:58:55.366326 20283 rpcz_store.cc:276] 0924 19:58:53.087618 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:53.111205 (+ 23587us) service_pool.cc:224] Handling call
0924 19:58:55.036438 (+1925233us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:55.365972 (+329534us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":3136,"Hive Metastore.run_cpu_time_us":369,"Hive Metastore.run_wall_time_us":179144}
I20260924 19:58:55.369753 21263 mvcc.cc:204] Tried to move back new op lower bound from 7332986602160685056 to 7332986570936090624. Current Snapshot: MvccSnapshot[applied={T|T < 7332986602160685056 or (T in {7332986602160685056})}]
I20260924 19:58:55.396871 20283 master_service.cc:960] ReplaceTablet: received request to replace tablet 811a1580acf747b49d8db8e2ae38b1f4 from {username='slave'} at 127.0.0.1:40798
W20260924 19:58:55.419052 20283 catalog_manager.cc:5671] Ignoring report from unknown tablet a623265a78374bb2a3dc20851ae4aaa2
I20260924 19:58:55.435582 21130 consensus_queue.cc:1048] T a623265a78374bb2a3dc20851ae4aaa2 P 0055218dbcbd48b58018e9774e59e14c [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:55.442792 20283 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_6ac8d98b233146c495713e0cda0ac6b5" } new_table_name: "default.Table_392dba2c1022441bb894d49994624e8d" modify_external_catalogs: true
I20260924 19:58:55.465920 20281 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_c8b8f1cd14774827a2cc045e0ffbf35d" } new_table_name: "default.Table_0b0df5bd140e47a5ac7a3c25f99a0ace" modify_external_catalogs: true
I20260924 19:58:55.534703 20279 master_service.cc:960] ReplaceTablet: received request to replace tablet 811a1580acf747b49d8db8e2ae38b1f4 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:55.571316 20279 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_9067ff1ac3de431993245b7bafa4a8df" } modify_external_catalogs: true
I20260924 19:58:55.571947 20279 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40798:
table { table_name: "default.Table_9067ff1ac3de431993245b7bafa4a8df" } modify_external_catalogs: true
I20260924 19:58:55.607472 20276 master_service.cc:960] ReplaceTablet: received request to replace tablet 635f4961769f49dbb908af7904d29042 from {username='slave'} at 127.0.0.1:40798
I20260924 19:58:55.724284 20275 catalog_manager.cc:2770] deleted HMS catalog entry for table default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]
I20260924 19:58:55.724808 17418 master-stress-test.cc:579] Killing test thread 1/10
W20260924 19:58:55.739708 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 0.574s (SENT)
I20260924 19:58:55.740208 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:55.757210 21348 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260924 19:58:55.880455 21354 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f09f90f3506b48cd9c797dbbeffd0721)
I20260924 19:58:55.881108 21354 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:55.884738 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 3 candidate_status { last_received { term: 2 index: 129 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721" is_pre_election: true
I20260924 19:58:55.887756 20284 catalog_manager.cc:2213] created HMS catalog entry for table [id=63b88f84ea084d86a1aee7e14b122265]
I20260924 19:58:55.917635 21354 leader_election.cc:290] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
W20260924 19:58:55.928936 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 115ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:58:55.967896 21359 consensus_queue.cc:1048] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:56.006441 21194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 3 candidate_status { last_received { term: 2 index: 129 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934" is_pre_election: true
I20260924 19:58:56.007792 21194 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1feab783ba9d4112932c72545e55ea19 in term 2.
I20260924 19:58:56.011375 21359 consensus_queue.cc:1048] T 811a1580acf747b49d8db8e2ae38b1f4 P af20c71727a14e398a217d4623d1399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:56.011500 20196 leader_election.cc:304] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934, 1feab783ba9d4112932c72545e55ea19; no voters: f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:56.012697 21354 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 19:58:56.013046 21354 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f09f90f3506b48cd9c797dbbeffd0721)
I20260924 19:58:56.013377 21354 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:58:56.037357 21354 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:56.052572 20278 catalog_manager.cc:2213] created HMS catalog entry for table [id=0cd83cf912964f9f8491eee44789d366]
W20260924 19:58:56.060952 20276 catalog_manager.cc:5671] Ignoring report from unknown tablet 811a1580acf747b49d8db8e2ae38b1f4
I20260924 19:58:56.083675 20218 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 3 FOLLOWER]: Rejecting Update request from peer f09f90f3506b48cd9c797dbbeffd0721 for earlier term 2. Current term is 3. Ops: [2.130-2.130]
I20260924 19:58:56.086313 21362 consensus_queue.cc:1059] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 }, Status: INVALID_TERM, Last received: 2.129, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20260924 19:58:56.086881 20294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 3 candidate_status { last_received { term: 2 index: 129 } } ignore_live_leader: false dest_uuid: "f09f90f3506b48cd9c797dbbeffd0721"
I20260924 19:58:56.094568 21194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1feab783ba9d4112932c72545e55ea19" candidate_term: 3 candidate_status { last_received { term: 2 index: 129 } } ignore_live_leader: false dest_uuid: "022c1898fcbe4856aa680c26047f6934"
I20260924 19:58:56.095098 21194 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:58:56.103652 21194 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1feab783ba9d4112932c72545e55ea19 in term 3.
I20260924 19:58:56.104336 21354 leader_election.cc:290] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 3 election: Requested vote from peers f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
I20260924 19:58:56.106184 20196 leader_election.cc:304] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934, 1feab783ba9d4112932c72545e55ea19; no voters: f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:56.108449 21362 raft_consensus.cc:3055] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 2 LEADER]: Stepping down as leader of term 2
I20260924 19:58:56.109383 21362 raft_consensus.cc:740] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f09f90f3506b48cd9c797dbbeffd0721, State: Running, Role: LEADER
I20260924 19:58:56.110407 21354 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:58:56.111471 21354 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [term 3 LEADER]: Becoming Leader. State: Replica: 1feab783ba9d4112932c72545e55ea19, State: Running, Role: LEADER
I20260924 19:58:56.111093 21362 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 2.132, Last appended by leader: 132, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:56.113827 21362 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:58:56.115964 21354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 2.129, Last appended by leader: 129, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:56.130228 21370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1feab783ba9d4112932c72545e55ea19. Latest consensus state: current_term: 3 leader_uuid: "1feab783ba9d4112932c72545e55ea19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:56.137838 21370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: This master's current role is: LEADER
I20260924 19:58:56.149082 20282 catalog_manager.cc:2213] created HMS catalog entry for table [id=a72b2daafffe4b468b98d480adb51036]
W20260924 19:58:56.151603 20282 catalog_manager.cc:2241] Illegal state: an error occurred while writing to the sys-catalog: replica f09f90f3506b48cd9c797dbbeffd0721 is not leader of this config: current role FOLLOWER
W20260924 19:58:56.153443 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:58:56.156279 20268 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 consensus state has no leader
I20260924 19:58:56.162446 21371 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260924 19:58:56.190533 21371 catalog_manager.cc:692] Loaded metadata for table default.table_5edadeb38f0143aea238c6775f09b6e5 [id=3395dfcca2e84761aa9f9ec23fbd10cf]
I20260924 19:58:56.192559 21371 catalog_manager.cc:692] Loaded metadata for table default.table_1c56d0ae4f73481fab5eea13c17efd81 [id=3914fcd32f4e4eea934bfb6b273de449]
I20260924 19:58:56.200940 21371 catalog_manager.cc:692] Loaded metadata for table default.table_6ac8d98b233146c495713e0cda0ac6b5 [id=46a1524a039b484d9118dee4b62cb8f9]
I20260924 19:58:56.202889 21371 catalog_manager.cc:692] Loaded metadata for table default.table_c8b8f1cd14774827a2cc045e0ffbf35d [id=966de0fe489d49ea9ae3954d0d0916c0]
I20260924 19:58:56.206857 20277 catalog_manager.cc:2213] created HMS catalog entry for table [id=990c22a366c749ad8b0c66ac2e71b273]
W20260924 19:58:56.208950 20277 catalog_manager.cc:2241] Illegal state: an error occurred while writing to the sys-catalog: replica f09f90f3506b48cd9c797dbbeffd0721 is not leader of this config: current role FOLLOWER
I20260924 19:58:56.215744 21371 catalog_manager.cc:692] Loaded metadata for table default.table_2a7727ff525b44bc97232bb07f824105 [id=99cc6ce05d86411d93e5afc71ae06fb2]
I20260924 19:58:56.217419 20438 catalog_manager.cc:2811] Deleting table default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300] in response to Hive Metastore notification log event 23
I20260924 19:58:56.217657 21371 catalog_manager.cc:692] Loaded metadata for table default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300]
I20260924 19:58:56.219638 21371 catalog_manager.cc:692] Loaded metadata for table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1]
I20260924 19:58:56.221262 21371 catalog_manager.cc:692] Loaded metadata for table default.table_9067ff1ac3de431993245b7bafa4a8df [id=c2d80fec62674432b58189903dc8e9d2]
W20260924 19:58:56.223199 20438 catalog_manager.cc:2887] Illegal state: an error occurred while updating the sys-catalog: replica f09f90f3506b48cd9c797dbbeffd0721 is not leader of this config: current role FOLLOWER
W20260924 19:58:56.223794 20438 hms_notification_log_listener.cc:330] Failed to handle Hive Metastore notification: 23 DROP_TABLE default.table_f25b80aa63f9428d849b3723d9f25a39: Illegal state: an error occurred while updating the sys-catalog: replica f09f90f3506b48cd9c797dbbeffd0721 is not leader of this config: current role FOLLOWER
W20260924 19:58:56.224319 20438 hms_notification_log_listener.cc:143] Hive Metastore notification log listener poll failed: Service unavailable: lost leadership while handling Hive Metastore notification log events: an error occurred while updating the sys-catalog: replica f09f90f3506b48cd9c797dbbeffd0721 is not leader of this config: current role FOLLOWER
I20260924 19:58:56.229849 21371 catalog_manager.cc:692] Loaded metadata for table default.table_b9baeba47f2642b38898c5b2f045cccf [id=c76d82558bd147a6b85aed1a31ef4f49]
W20260924 19:58:56.234784 20925 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.17.2.189:40117)
I20260924 19:58:56.247048 20275 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40798 (request call id 86) took 2811 ms. Trace:
I20260924 19:58:56.247402 20275 rpcz_store.cc:276] 0924 19:58:53.435952 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:53.514612 (+ 78660us) service_pool.cc:224] Handling call
0924 19:58:55.223381 (+1708769us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:56.246984 (+1023603us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":50427,"Hive Metastore.run_cpu_time_us":321,"Hive Metastore.run_wall_time_us":449175}
I20260924 19:58:56.268266 21371 tablet_loader.cc:96] loaded metadata for tablet 3bcd130dba3e4574949f60afe7474fc2 (table default.table_9067ff1ac3de431993245b7bafa4a8df [id=c2d80fec62674432b58189903dc8e9d2])
I20260924 19:58:56.269690 21371 tablet_loader.cc:96] loaded metadata for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1])
I20260924 19:58:56.275965 21371 tablet_loader.cc:96] loaded metadata for tablet 5d2020b2b1f7446e8526eaabd7c2e8fb (table default.table_1c56d0ae4f73481fab5eea13c17efd81 [id=3914fcd32f4e4eea934bfb6b273de449])
I20260924 19:58:56.277525 21371 tablet_loader.cc:96] loaded metadata for tablet 5fa3110891954347a1cff0c06a23de63 (table default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300])
I20260924 19:58:56.278823 21371 tablet_loader.cc:96] loaded metadata for tablet 7a22f69f014340b0a9d775c4d6688e21 (table default.table_2a7727ff525b44bc97232bb07f824105 [id=99cc6ce05d86411d93e5afc71ae06fb2])
I20260924 19:58:56.280150 21371 tablet_loader.cc:96] loaded metadata for tablet ac71ad6f430d4bfab1cdb6ec6163e656 (table default.table_c8b8f1cd14774827a2cc045e0ffbf35d [id=966de0fe489d49ea9ae3954d0d0916c0])
I20260924 19:58:56.281450 21371 tablet_loader.cc:96] loaded metadata for tablet af1b7b9f6aa24acbbdcf37b03fa65597 (table default.table_b9baeba47f2642b38898c5b2f045cccf [id=c76d82558bd147a6b85aed1a31ef4f49])
I20260924 19:58:56.282804 21371 tablet_loader.cc:96] loaded metadata for tablet ba5cf6b97298401dac6aea045b4ebb47 (table default.table_5edadeb38f0143aea238c6775f09b6e5 [id=3395dfcca2e84761aa9f9ec23fbd10cf])
I20260924 19:58:56.299052 21371 tablet_loader.cc:96] loaded metadata for tablet f050831a13f24241a9f314545d1f5457 (table default.table_6ac8d98b233146c495713e0cda0ac6b5 [id=46a1524a039b484d9118dee4b62cb8f9])
I20260924 19:58:56.302233 21371 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260924 19:58:56.315420 21371 catalog_manager.cc:1308] Loaded cluster ID: 543042799ec94022838cc92e16f8796f
I20260924 19:58:56.315702 21371 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260924 19:58:56.332818 21371 catalog_manager.cc:1553] Loading token signing keys...
I20260924 19:58:56.345512 21371 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19: Loaded TSK: 0
I20260924 19:58:56.351147 21371 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260924 19:58:56.377924 21371 catalog_manager.cc:1576] Loading latest processed Hive Metastore notification log event ID...
I20260924 19:58:56.407549 21371 catalog_manager.cc:6068] Last processed Hive Metastore notification event ID: 22
I20260924 19:58:56.418430 20282 catalog_manager.cc:2223] deleted orphaned HMS catalog entry for table [id=a72b2daafffe4b468b98d480adb51036]
I20260924 19:58:56.423983 20282 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 99) took 1586 ms. Trace:
I20260924 19:58:56.424338 20282 rpcz_store.cc:276] 0924 19:58:54.838084 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:54.838266 (+ 182us) service_pool.cc:224] Handling call
0924 19:58:55.262736 (+424470us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:55.269734 (+ 6998us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:56.149033 (+879299us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:56.151289 (+ 2256us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:56.151326 (+ 37us) write_op.cc:454] Released schema lock
0924 19:58:56.151947 (+ 621us) catalog_manager.cc:2220] Rolling back HMS table creation
0924 19:58:56.423913 (+271966us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:56.150965 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica f09f90f3506b48cd9c797dbbeffd0721 is not leader of this config: current role FOLLOWER)
0924 19:58:56.151022 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:56.151045 (+ 23us) write_op.cc:454] Released schema lock
0924 19:58:56.151069 (+ 24us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":838896,"Hive Metastore.run_cpu_time_us":1015,"Hive Metastore.run_wall_time_us":303377,"child_traces":[["op",{}]]}
W20260924 19:58:56.428134 20925 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.2.190:44951)
I20260924 19:58:56.502013 20294 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 3 FOLLOWER]: Refusing update from remote peer 1feab783ba9d4112932c72545e55ea19: Log matching property violated. Preceding OpId in replica: term: 2 index: 132. Preceding OpId from leader: term: 3 index: 131. (term mismatch)
I20260924 19:58:56.502388 20294 pending_rounds.cc:85] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Aborting all ops after (but not including) 130
I20260924 19:58:56.502666 20294 pending_rounds.cc:107] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Aborting uncommitted WRITE_OP operation due to leader change: 2.131
W20260924 19:58:56.503772 20440 catalog_manager.cc:6644] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20260924 19:58:56.503908 20294 pending_rounds.cc:107] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Aborting uncommitted WRITE_OP operation due to leader change: 2.132
E20260924 19:58:56.504048 20440 catalog_manager.cc:859] Error processing tables/tablets deletions: Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20260924 19:58:56.508486 21194 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Refusing update from remote peer 1feab783ba9d4112932c72545e55ea19: Log matching property violated. Preceding OpId in replica: term: 2 index: 106. Preceding OpId from leader: term: 3 index: 131. (index mismatch)
I20260924 19:58:56.510946 21370 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 106, Time since last communication: 0.000s
W20260924 19:58:56.518373 20278 catalog_manager.cc:2241] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20260924 19:58:56.521507 21354 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20260924 19:58:56.522271 21354 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [LEADER]: Peer f09f90f3506b48cd9c797dbbeffd0721 log is divergent from this leader: its last log entry 2.130 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 129
I20260924 19:58:56.525100 20294 pending_rounds.cc:85] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Aborting all ops after (but not including) 129
I20260924 19:58:56.531278 20294 pending_rounds.cc:107] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721: Aborting uncommitted WRITE_OP operation due to leader change: 2.130
W20260924 19:58:56.532356 20284 catalog_manager.cc:2241] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20260924 19:58:56.539965 21362 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1feab783ba9d4112932c72545e55ea19. Latest consensus state: current_term: 3 leader_uuid: "1feab783ba9d4112932c72545e55ea19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:56.540676 21362 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:56.434551 20210 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42838:
name: "default.Table_9a44188ed4234e4e98ebdcb2e803b418"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:56.420629 20208 catalog_manager.cc:3583] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42838:
table { table_name: "default.Table_6ac8d98b233146c495713e0cda0ac6b5" } new_table_name: "default.Table_392dba2c1022441bb894d49994624e8d" modify_external_catalogs: true
I20260924 19:58:56.551571 21356 mvcc.cc:204] Tried to move back new op lower bound from 7332986619855343616 to 7332986618347094016. Current Snapshot: MvccSnapshot[applied={T|T < 7332986619855343616}]
I20260924 19:58:56.573009 20471 catalog_manager.cc:2811] Deleting table default.table_f25b80aa63f9428d849b3723d9f25a39 [id=a41c0a152a10436fb22b0c1b14abd300] in response to Hive Metastore notification log event 23
I20260924 19:58:56.585515 21370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "1feab783ba9d4112932c72545e55ea19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:56.586380 21370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: This master's current role is: LEADER
I20260924 19:58:56.619087 21362 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1feab783ba9d4112932c72545e55ea19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:56.619977 21362 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:56.663112 20471 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19: Sending DeleteTablet for 3 replicas of tablet 5fa3110891954347a1cff0c06a23de63
I20260924 19:58:56.663475 20915 heartbeater.cc:499] Master 127.17.2.190:44951 was elected leader, sending a full tablet report...
W20260924 19:58:56.696658 20209 catalog_manager.cc:5671] Ignoring report from unknown tablet a623265a78374bb2a3dc20851ae4aaa2
W20260924 19:58:56.696975 20209 catalog_manager.cc:5671] Ignoring report from unknown tablet 0d49e3b1113b4614b359e9922e7fe55f
W20260924 19:58:56.697175 20209 catalog_manager.cc:5671] Ignoring report from unknown tablet 65e0685bc4564703b88a5247511686ec
W20260924 19:58:56.697353 20209 catalog_manager.cc:5671] Ignoring report from unknown tablet 811a1580acf747b49d8db8e2ae38b1f4
W20260924 19:58:56.697538 20209 catalog_manager.cc:5671] Ignoring report from unknown tablet c54e96fe673f438eb2919be2ad785641
W20260924 19:58:56.697719 20209 catalog_manager.cc:5671] Ignoring report from unknown tablet e2719ada60a143c6862e90d81f9c2d82
W20260924 19:58:56.697913 20209 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
W20260924 19:58:56.703125 20804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:33667, user_credentials={real_user=slave}} blocked reactor thread for 55704.7us
W20260924 19:58:56.712661 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:58:56.753003 20553 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fa3110891954347a1cff0c06a23de63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:56 UTC) from {username='slave'} at 127.0.0.1:52098
I20260924 19:58:56.793303 20471 catalog_manager.cc:2811] Deleting table default.table_9a44188ed4234e4e98ebdcb2e803b418 [id=a72b2daafffe4b468b98d480adb51036] in response to Hive Metastore notification log event 28
W20260924 19:58:56.794279 20471 hms_notification_log_listener.cc:330] Failed to handle Hive Metastore notification: 28 DROP_TABLE default.table_9a44188ed4234e4e98ebdcb2e803b418: Not found: the table does not exist: table_id: "a72b2daafffe4b468b98d480adb51036" table_name: "default.table_9a44188ed4234e4e98ebdcb2e803b418"
I20260924 19:58:56.805464 20701 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fa3110891954347a1cff0c06a23de63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:56 UTC) from {username='slave'} at 127.0.0.1:37446
W20260924 19:58:56.806665 20197 catalog_manager.cc:5030] TS 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): delete failed for tablet 5fa3110891954347a1cff0c06a23de63 because the tablet was not found. No further retry: Not found: Tablet not found: 5fa3110891954347a1cff0c06a23de63
I20260924 19:58:56.813674 20846 tablet_service.cc:1558] Processing DeleteTablet for tablet 5fa3110891954347a1cff0c06a23de63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:58:56 UTC) from {username='slave'} at 127.0.0.1:45790
W20260924 19:58:56.822435 20210 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9a44188ed4234e4e98ebdcb2e803b418 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260924 19:58:56.827517 20197 catalog_manager.cc:5030] TS af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): delete failed for tablet 5fa3110891954347a1cff0c06a23de63 because the tablet was not found. No further retry: Not found: Tablet not found: 5fa3110891954347a1cff0c06a23de63
W20260924 19:58:56.853597 20196 catalog_manager.cc:5030] TS 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): delete failed for tablet 5fa3110891954347a1cff0c06a23de63 because the tablet was not found. No further retry: Not found: Tablet not found: 5fa3110891954347a1cff0c06a23de63
I20260924 19:58:56.861132 20277 catalog_manager.cc:2223] deleted orphaned HMS catalog entry for table [id=990c22a366c749ad8b0c66ac2e71b273]
I20260924 19:58:56.862000 20277 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 103) took 1727 ms. Trace:
I20260924 19:58:56.862291 20277 rpcz_store.cc:276] 0924 19:58:55.134772 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:55.137829 (+ 3057us) service_pool.cc:224] Handling call
0924 19:58:55.146454 (+ 8625us) spinlock_profiling.cc:243] Waited 6.16 ms on lock 0x7b54000104d4. stack: 00007fe7ffab1323 00007fe7ffab1220 00007fe7ff655574 0000000000516964 00000000005168f9 00007fe806baf5c0 00007fe806c4e6fa 00007fe8089be98c 00007fe808abe166 00007fe80666a034 00007fe806669fc2 00007fe806669ef1 00007fe806669e6c 00007fe806669122 00007fe80491409c 00007fe8049134d6
0924 19:58:55.364334 (+217880us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:55.376531 (+ 12197us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:56.206814 (+830283us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:56.208692 (+ 1878us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:56.208715 (+ 23us) write_op.cc:454] Released schema lock
0924 19:58:56.209256 (+ 541us) catalog_manager.cc:2220] Rolling back HMS table creation
0924 19:58:56.861935 (+652679us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:56.208471 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica f09f90f3506b48cd9c797dbbeffd0721 is not leader of this config: current role FOLLOWER)
0924 19:58:56.208507 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:56.208530 (+ 23us) write_op.cc:454] Released schema lock
0924 19:58:56.208552 (+ 22us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":978440,"Hive Metastore.run_cpu_time_us":1016,"Hive Metastore.run_wall_time_us":502645,"spinlock_wait_cycles":14168960,"child_traces":[["op",{}]]}
I20260924 19:58:56.864552 20210 catalog_manager.cc:2213] created HMS catalog entry for table [id=05b08e63badd4e81a724aa6c1fe049cc]
I20260924 19:58:56.875783 20209 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42838:
name: "default.Table_d876eec98f7a4157b29a3241d303ab96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:56.906896 20471 catalog_manager.cc:2811] Deleting table default.table_d876eec98f7a4157b29a3241d303ab96 [id=990c22a366c749ad8b0c66ac2e71b273] in response to Hive Metastore notification log event 29
W20260924 19:58:56.908104 20471 hms_notification_log_listener.cc:330] Failed to handle Hive Metastore notification: 29 DROP_TABLE default.table_d876eec98f7a4157b29a3241d303ab96: Not found: the table does not exist: table_id: "990c22a366c749ad8b0c66ac2e71b273" table_name: "default.table_d876eec98f7a4157b29a3241d303ab96"
I20260924 19:58:56.918462 21374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1feab783ba9d4112932c72545e55ea19. Latest consensus state: current_term: 3 leader_uuid: "1feab783ba9d4112932c72545e55ea19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:56.919255 21374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:57.002041 21354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "1feab783ba9d4112932c72545e55ea19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:57.004396 21354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1feab783ba9d4112932c72545e55ea19 [sys.catalog]: This master's current role is: LEADER
I20260924 19:58:57.030503 21374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1feab783ba9d4112932c72545e55ea19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } } }
I20260924 19:58:57.031224 21374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:58:57.033806 20771 heartbeater.cc:499] Master 127.17.2.190:44951 was elected leader, sending a full tablet report...
W20260924 19:58:57.042232 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet a623265a78374bb2a3dc20851ae4aaa2
W20260924 19:58:57.042482 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 19:58:57.042659 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet e2719ada60a143c6862e90d81f9c2d82
W20260924 19:58:57.042831 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet 65e0685bc4564703b88a5247511686ec
W20260924 19:58:57.043002 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet 811a1580acf747b49d8db8e2ae38b1f4
W20260924 19:58:57.048951 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet c54e96fe673f438eb2919be2ad785641
W20260924 19:58:57.049185 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:57.050191 20846 tablet_service.cc:1511] Processing CreateTablet for tablet 3086f72543394aa8ba09cf3566feee6c (DEFAULT_TABLE table=default.table_9a44188ed4234e4e98ebdcb2e803b418 [id=05b08e63badd4e81a724aa6c1fe049cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:57.051270 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3086f72543394aa8ba09cf3566feee6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:57.064438 20553 tablet_service.cc:1511] Processing CreateTablet for tablet 3086f72543394aa8ba09cf3566feee6c (DEFAULT_TABLE table=default.table_9a44188ed4234e4e98ebdcb2e803b418 [id=05b08e63badd4e81a724aa6c1fe049cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:57.066036 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3086f72543394aa8ba09cf3566feee6c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:58:57.080127 20658 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.129:33667, user_credentials={real_user=slave}} blocked reactor thread for 57336.2us
I20260924 19:58:57.084544 20701 tablet_service.cc:1511] Processing CreateTablet for tablet 3086f72543394aa8ba09cf3566feee6c (DEFAULT_TABLE table=default.table_9a44188ed4234e4e98ebdcb2e803b418 [id=05b08e63badd4e81a724aa6c1fe049cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:57.085570 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3086f72543394aa8ba09cf3566feee6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:57.104883 20622 heartbeater.cc:499] Master 127.17.2.190:44951 was elected leader, sending a full tablet report...
I20260924 19:58:57.111205 21401 tablet_bootstrap.cc:492] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
W20260924 19:58:57.113329 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet a623265a78374bb2a3dc20851ae4aaa2
W20260924 19:58:57.113607 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet e2719ada60a143c6862e90d81f9c2d82
W20260924 19:58:57.113782 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet 65e0685bc4564703b88a5247511686ec
W20260924 19:58:57.113953 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet 811a1580acf747b49d8db8e2ae38b1f4
W20260924 19:58:57.114120 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet c54e96fe673f438eb2919be2ad785641
W20260924 19:58:57.114308 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet 0c40348984314595b6e6e49c4f03f35e
W20260924 19:58:57.114502 20210 catalog_manager.cc:5671] Ignoring report from unknown tablet 635f4961769f49dbb908af7904d29042
I20260924 19:58:57.137867 21401 tablet_bootstrap.cc:654] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:57.148216 21402 tablet_bootstrap.cc:492] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:57.151255 21403 tablet_bootstrap.cc:492] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:57.171319 21402 tablet_bootstrap.cc:654] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:57.174140 21401 tablet_bootstrap.cc:492] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:57.174677 21401 ts_tablet_manager.cc:1403] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.007s
I20260924 19:58:57.179440 21401 raft_consensus.cc:359] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:57.180305 21403 tablet_bootstrap.cc:654] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:57.180620 21401 raft_consensus.cc:385] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:57.185000 21401 raft_consensus.cc:740] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
W20260924 19:58:57.186131 20268 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 consensus state has no leader
I20260924 19:58:57.185956 21401 consensus_queue.cc:260] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:57.189148 21402 tablet_bootstrap.cc:492] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:57.189613 21402 ts_tablet_manager.cc:1403] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.000s
I20260924 19:58:57.192534 21402 raft_consensus.cc:359] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:57.193234 21402 raft_consensus.cc:385] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:57.193519 21402 raft_consensus.cc:740] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:57.201220 21401 ts_tablet_manager.cc:1434] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260924 19:58:57.194222 21402 consensus_queue.cc:260] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:57.213353 21402 ts_tablet_manager.cc:1434] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260924 19:58:57.220651 21403 tablet_bootstrap.cc:492] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:57.221108 21403 ts_tablet_manager.cc:1403] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.010s
I20260924 19:58:57.241268 21403 raft_consensus.cc:359] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:57.242195 21403 raft_consensus.cc:385] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:57.242471 21403 raft_consensus.cc:740] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
W20260924 19:58:57.243276 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:58:57.243178 21403 consensus_queue.cc:260] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:57.245705 21403 ts_tablet_manager.cc:1434] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.024s user 0.002s sys 0.004s
I20260924 19:58:57.381695 21399 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:57.382192 21399 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:57.384290 21399 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:58:57.387462 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 19:58:57.429661 20509 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.17.2.130:38719, user_credentials={real_user=slave}} blocked reactor thread for 46148.6us
W20260924 19:58:57.432036 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:58:57.432456 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:58:57.433537 21399 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:58:57.470211 21399 raft_consensus.cc:493] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:57.470796 21399 raft_consensus.cc:515] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
W20260924 19:58:57.472337 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 2154ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:58:57.473059 21399 leader_election.cc:290] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601), 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667)
I20260924 19:58:57.474450 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3086f72543394aa8ba09cf3566feee6c" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:57.474689 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3086f72543394aa8ba09cf3566feee6c" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:57.475181 20574 raft_consensus.cc:2468] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:57.475275 20867 raft_consensus.cc:2468] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:57.476466 20657 leader_election.cc:304] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:57.478276 21399 raft_consensus.cc:2804] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:57.478627 21399 raft_consensus.cc:493] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:57.478951 21399 raft_consensus.cc:3060] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:57.486284 21399 raft_consensus.cc:515] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:57.489157 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3086f72543394aa8ba09cf3566feee6c" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:57.489656 20867 raft_consensus.cc:3060] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:57.495709 20867 raft_consensus.cc:2468] T 3086f72543394aa8ba09cf3566feee6c P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:57.497004 20657 leader_election.cc:304] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:57.497802 21400 raft_consensus.cc:2804] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:57.498271 21400 raft_consensus.cc:697] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Becoming Leader. State: Replica: af20c71727a14e398a217d4623d1399a, State: Running, Role: LEADER
I20260924 19:58:57.499032 21400 consensus_queue.cc:237] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } }
I20260924 19:58:57.501350 21399 leader_election.cc:290] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Requested vote from peers 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601), 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667)
I20260924 19:58:57.502836 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3086f72543394aa8ba09cf3566feee6c" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:57.503549 20574 raft_consensus.cc:3060] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:57.512373 20574 raft_consensus.cc:2468] T 3086f72543394aa8ba09cf3566feee6c P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:57.542126 20207 catalog_manager.cc:5838] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a reported cstate change: term changed from 0 to 1, leader changed from <none> to af20c71727a14e398a217d4623d1399a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "af20c71727a14e398a217d4623d1399a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: HEALTHY } } }
W20260924 19:58:57.673787 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:58:57.722152 20208 catalog_manager.cc:3541] renamed table default.table_6ac8d98b233146c495713e0cda0ac6b5 [id=46a1524a039b484d9118dee4b62cb8f9] in HMS: new name default.table_392dba2c1022441bb894d49994624e8d
I20260924 19:58:57.732568 20281 catalog_manager.cc:3541] renamed table default.table_c8b8f1cd14774827a2cc045e0ffbf35d [id=966de0fe489d49ea9ae3954d0d0916c0] in HMS: new name default.table_0b0df5bd140e47a5ac7a3c25f99a0ace
W20260924 19:58:57.732919 20209 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d876eec98f7a4157b29a3241d303ab96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:58:57.736094 20281 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40798 (request call id 108) took 2270 ms. Trace:
I20260924 19:58:57.736464 20281 rpcz_store.cc:276] 0924 19:58:55.465385 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:55.465534 (+ 149us) service_pool.cc:224] Handling call
0924 19:58:56.245085 (+779551us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:57.733995 (+1488910us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:57.735012 (+ 1017us) catalog_manager.cc:3764] Apply alter schema
0924 19:58:57.736021 (+ 1009us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":615103,"Hive Metastore.run_cpu_time_us":1881,"Hive Metastore.run_wall_time_us":869139}
F20260924 19:58:57.739749 20471 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 31, previous 31; dumping first 2 out of 2 received events: NotificationEvent(eventId=31, eventTime=1790279937, eventType=ALTER_TABLE, dbName=default, tableName=table_0b0df5bd140e47a5ac7a3c25f99a0ace, message={"server":"","servicePrincipal":"","db":"default","table":"table_c8b8f1cd14774827a2cc045e0ffbf35d","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_c8b8f1cd14774827a2cc045e0ffbf35d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1790279925},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/warehouse/table_c8b8f1cd14774827a2cc045e0ffbf35d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_c8b8f1cd14774827a2cc045e0ffbf35d\",\"kudu.table_id\":\"966de0fe489d49ea9ae3954d0d0916c0\",\"transient_lastDdlTime\":\"1790279925\",\"comment\":\"\",\"kudu.master_addresses\":\"127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"543042799ec94022838cc92e16f8796f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_0b0df5bd140e47a5ac7a3c25f99a0ace\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1790279925},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/warehouse/table_0b0df5bd140e47a5ac7a3c25f99a0ace\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_0b0df5bd140e47a5ac7a3c25f99a0ace\",\"kudu.table_id\":\"966de0fe489d49ea9ae3954d0d0916c0\",\"transient_lastDdlTime\":\"1790279925\",\"comment\":\"\",\"kudu.master_addresses\":\"127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"543042799ec94022838cc92e16f8796f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1790279937}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=31, eventTime=1790279937, eventType=ALTER_TABLE, dbName=default, tableName=table_392dba2c1022441bb894d49994624e8d, message={"server":"","servicePrincipal":"","db":"default","table":"table_6ac8d98b233146c495713e0cda0ac6b5","tableType":"MANAGED_TABLE","tableObjBeforeJson":"{\"1\":{\"str\":\"table_6ac8d98b233146c495713e0cda0ac6b5\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1790279930},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/warehouse/table_6ac8d98b233146c495713e0cda0ac6b5\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_6ac8d98b233146c495713e0cda0ac6b5\",\"kudu.table_id\":\"46a1524a039b484d9118dee4b62cb8f9\",\"transient_lastDdlTime\":\"1790279930\",\"comment\":\"\",\"kudu.master_addresses\":\"127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"543042799ec94022838cc92e16f8796f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","tableObjAfterJson":"{\"1\":{\"str\":\"table_392dba2c1022441bb894d49994624e8d\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1790279930},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-task0eKU8U/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1790279826552838-17418-0/minicluster-data/warehouse/table_392dba2c1022441bb894d49994624e8d\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_392dba2c1022441bb894d49994624e8d\",\"kudu.table_id\":\"46a1524a039b484d9118dee4b62cb8f9\",\"transient_lastDdlTime\":\"1790279930\",\"comment\":\"\",\"kudu.master_addresses\":\"127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"543042799ec94022838cc92e16f8796f\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"15\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","isTruncateOp":"false","timestamp":1790279937}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260924 19:58:57.793283 20209 catalog_manager.cc:2213] created HMS catalog entry for table [id=33b7e3f8888943c786bf9d69d4b41f9d]
@ 0x7f12637b4920 google::LogMessage::Flush() at ??:0
I20260924 19:58:57.847409 20279 catalog_manager.cc:2770] deleted HMS catalog entry for table default.table_9067ff1ac3de431993245b7bafa4a8df [id=c2d80fec62674432b58189903dc8e9d2]
I20260924 19:58:57.848647 20279 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40798 (request call id 110) took 2278 ms. Trace:
I20260924 19:58:57.848949 20279 rpcz_store.cc:276] 0924 19:58:55.570781 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:55.570984 (+ 203us) service_pool.cc:224] Handling call
0924 19:58:56.246213 (+675229us) catalog_manager.cc:2432] Looking up, locking, and authorizing table
0924 19:58:57.848579 (+1602366us) inbound_call.cc:177] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1483621,"Hive Metastore.run_cpu_time_us":404,"Hive Metastore.run_wall_time_us":116922}
@ 0x7f12637b468f google::LogMessage::~LogMessage() at ??:0
I20260924 19:58:57.894165 20553 tablet_service.cc:1511] Processing CreateTablet for tablet 7da411d09de34ac88ce9e244b721fee5 (DEFAULT_TABLE table=default.table_d876eec98f7a4157b29a3241d303ab96 [id=33b7e3f8888943c786bf9d69d4b41f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:57.895284 20553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da411d09de34ac88ce9e244b721fee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:57.896304 20701 tablet_service.cc:1511] Processing CreateTablet for tablet 7da411d09de34ac88ce9e244b721fee5 (DEFAULT_TABLE table=default.table_d876eec98f7a4157b29a3241d303ab96 [id=33b7e3f8888943c786bf9d69d4b41f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:57.897390 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da411d09de34ac88ce9e244b721fee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:57.901958 20846 tablet_service.cc:1511] Processing CreateTablet for tablet 7da411d09de34ac88ce9e244b721fee5 (DEFAULT_TABLE table=default.table_d876eec98f7a4157b29a3241d303ab96 [id=33b7e3f8888943c786bf9d69d4b41f9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:58:57.902935 20846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da411d09de34ac88ce9e244b721fee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:58:57.929345 21414 tablet_bootstrap.cc:492] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a: Bootstrap starting.
I20260924 19:58:57.944329 21414 tablet_bootstrap.cc:654] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:57.957273 21416 tablet_bootstrap.cc:492] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622: Bootstrap starting.
I20260924 19:58:57.956738 21417 tablet_bootstrap.cc:492] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c: Bootstrap starting.
I20260924 19:58:57.963461 21414 tablet_bootstrap.cc:492] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a: No bootstrap required, opened a new log
I20260924 19:58:57.963980 21414 ts_tablet_manager.cc:1403] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.002s
I20260924 19:58:57.971237 21416 tablet_bootstrap.cc:654] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622: Neither blocks nor log segments found. Creating new log.
I20260924 19:58:57.966830 21414 raft_consensus.cc:359] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:57.973732 21414 raft_consensus.cc:385] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:57.974138 21414 raft_consensus.cc:740] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af20c71727a14e398a217d4623d1399a, State: Initialized, Role: FOLLOWER
I20260924 19:58:57.975449 21414 consensus_queue.cc:260] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
@ 0x7f126e945f5c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260924 19:58:58.000844 21414 ts_tablet_manager.cc:1434] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a: Time spent starting tablet: real 0.037s user 0.003s sys 0.002s
I20260924 19:58:58.003379 21417 tablet_bootstrap.cc:654] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c: Neither blocks nor log segments found. Creating new log.
W20260924 19:58:58.014719 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 2154ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:58:58.027107 20278 catalog_manager.cc:2223] deleted orphaned HMS catalog entry for table [id=0cd83cf912964f9f8491eee44789d366]
I20260924 19:58:58.028040 20278 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 97) took 3409 ms. Trace:
I20260924 19:58:58.031020 21417 tablet_bootstrap.cc:492] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c: No bootstrap required, opened a new log
I20260924 19:58:58.031467 21417 ts_tablet_manager.cc:1403] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.005s
I20260924 19:58:58.028386 20278 rpcz_store.cc:276] 0924 19:58:54.619136 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:54.619358 (+ 222us) service_pool.cc:224] Handling call
0924 19:58:55.258095 (+638737us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:55.265189 (+ 7094us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:56.052522 (+787333us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:56.518726 (+466204us) catalog_manager.cc:2220] Rolling back HMS table creation
0924 19:58:58.027972 (+1509246us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:56.080253 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:56.080756 (+ 503us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:56.080788 (+ 32us) write_op.cc:435] Acquired schema lock
0924 19:58:56.080807 (+ 19us) tablet.cc:662] Decoding operations
0924 19:58:56.081255 (+ 448us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:56.081299 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:56.081368 (+ 69us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:56.081597 (+ 229us) tablet.cc:701] Row locks acquired
0924 19:58:56.081610 (+ 13us) write_op.cc:260] PREPARE: finished
0924 19:58:56.081742 (+ 132us) write_op.cc:270] Start()
0924 19:58:56.081835 (+ 93us) write_op.cc:276] Timestamp: P: 1790279936081686 usec, L: 0
0924 19:58:56.081853 (+ 18us) op_driver.cc:348] REPLICATION: starting
0924 19:58:56.082411 (+ 558us) log.cc:844] Serialized 433 byte log entry
0924 19:58:56.504306 (+421895us) op_driver.cc:464] REPLICATION: finished
0924 19:58:56.504340 (+ 34us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0924 19:58:56.504366 (+ 26us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:56.504474 (+ 108us) write_op.cc:454] Released schema lock
0924 19:58:56.504512 (+ 38us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1949764,"Hive Metastore.run_cpu_time_us":1006,"Hive Metastore.run_wall_time_us":343145,"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":25921,"prepare.run_cpu_time_us":2289,"prepare.run_wall_time_us":2395,"replication_time_us":422321}]]}
I20260924 19:58:58.034130 21417 raft_consensus.cc:359] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:58.034842 21417 raft_consensus.cc:385] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260924 19:58:58.035643 20925 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.2.190:44951)
I20260924 19:58:58.035127 21417 raft_consensus.cc:740] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0055218dbcbd48b58018e9774e59e14c, State: Initialized, Role: FOLLOWER
I20260924 19:58:58.036556 21417 consensus_queue.cc:260] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:58.041035 21416 tablet_bootstrap.cc:492] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622: No bootstrap required, opened a new log
I20260924 19:58:58.042007 21416 ts_tablet_manager.cc:1403] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.009s
I20260924 19:58:58.045696 20207 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42838:
name: "default.Table_d7b4827d2d284d37b72f600e797d818d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:58:58.054538 21417 ts_tablet_manager.cc:1434] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c: Time spent starting tablet: real 0.023s user 0.002s sys 0.004s
I20260924 19:58:58.070149 21416 raft_consensus.cc:359] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:58.078612 21416 raft_consensus.cc:385] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:58:58.082129 21416 raft_consensus.cc:740] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a4f50fc28c4616ba622ba51b0b2622, State: Initialized, Role: FOLLOWER
@ 0x7f126e94571c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260924 19:58:58.090867 21416 consensus_queue.cc:260] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:58.127645 21416 ts_tablet_manager.cc:1434] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622: Time spent starting tablet: real 0.085s user 0.007s sys 0.005s
W20260924 19:58:58.130815 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
@ 0x7f126e949792 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260924 19:58:58.247404 20284 catalog_manager.cc:2223] deleted orphaned HMS catalog entry for table [id=63b88f84ea084d86a1aee7e14b122265]
I20260924 19:58:58.248205 20284 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40798 (request call id 95) took 3774 ms. Trace:
I20260924 19:58:58.248502 20284 rpcz_store.cc:276] 0924 19:58:54.474080 (+ 0us) service_pool.cc:167] Inserting onto call queue
0924 19:58:54.507397 (+ 33317us) service_pool.cc:224] Handling call
0924 19:58:55.228426 (+721029us) catalog_manager.cc:2140] Acquired catalog manager lock
0924 19:58:55.244412 (+ 15986us) catalog_manager.cc:2186] Created new table and tablet info
0924 19:58:55.887709 (+643297us) catalog_manager.cc:2212] Created new table in HMS catalog
0924 19:58:56.532638 (+644929us) catalog_manager.cc:2220] Rolling back HMS table creation
0924 19:58:58.248138 (+1715500us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0924 19:58:55.965020 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0924 19:58:55.965524 (+ 504us) write_op.cc:432] Acquiring schema lock in shared mode
0924 19:58:55.965552 (+ 28us) write_op.cc:435] Acquired schema lock
0924 19:58:55.965569 (+ 17us) tablet.cc:662] Decoding operations
0924 19:58:55.965874 (+ 305us) write_op.cc:620] Acquiring the partition lock for write op
0924 19:58:55.965930 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0924 19:58:55.965961 (+ 31us) tablet.cc:685] Acquiring locks for 2 operations
0924 19:58:55.966187 (+ 226us) tablet.cc:701] Row locks acquired
0924 19:58:55.966203 (+ 16us) write_op.cc:260] PREPARE: finished
0924 19:58:55.966357 (+ 154us) write_op.cc:270] Start()
0924 19:58:55.966460 (+ 103us) write_op.cc:276] Timestamp: P: 1790279935966293 usec, L: 0
0924 19:58:55.966476 (+ 16us) op_driver.cc:348] REPLICATION: starting
0924 19:58:55.967083 (+ 607us) log.cc:844] Serialized 433 byte log entry
0924 19:58:56.531896 (+564813us) op_driver.cc:464] REPLICATION: finished
0924 19:58:56.531937 (+ 41us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0924 19:58:56.531968 (+ 31us) write_op.cc:489] Releasing partition, row and schema locks
0924 19:58:56.532102 (+ 134us) write_op.cc:454] Released schema lock
0924 19:58:56.532143 (+ 41us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1968676,"Hive Metastore.run_cpu_time_us":966,"Hive Metastore.run_wall_time_us":383635,"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":75657,"prepare.run_cpu_time_us":3010,"prepare.run_wall_time_us":61567,"raft.queue_time_us":32976,"raft.run_cpu_time_us":805,"raft.run_wall_time_us":1068,"replication_time_us":565242,"thread_start_us":135719,"threads_started":2}]]}
I20260924 19:58:58.261775 20209 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42838:
name: "default.Table_b9502ead412146d497587b0c995526fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f126e94974a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260924 19:58:58.284974 21400 consensus_queue.cc:1048] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:58.352146 21336 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:58.352782 21336 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:58.357781 21336 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 19:58:58.359091 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 19:58:58.361263 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 19:58:58.361655 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:58:58.362342 21336 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:58:58.368649 21426 consensus_queue.cc:1048] T 3086f72543394aa8ba09cf3566feee6c P af20c71727a14e398a217d4623d1399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
@ 0x7f126e9496da std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260924 19:58:58.457130 21434 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 19:58:58.457669 21434 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
@ 0x7f126e9496a2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260924 19:58:58.479750 20218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "022c1898fcbe4856aa680c26047f6934" candidate_term: 4 candidate_status { last_received { term: 3 index: 142 } } ignore_live_leader: false dest_uuid: "1feab783ba9d4112932c72545e55ea19" is_pre_election: true
I20260924 19:58:58.481582 21434 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
I20260924 19:58:58.525879 21400 raft_consensus.cc:493] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:58:58.526357 21400 raft_consensus.cc:515] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:58.528373 21400 leader_election.cc:290] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:58.529284 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7da411d09de34ac88ce9e244b721fee5" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622" is_pre_election: true
I20260924 19:58:58.529919 20574 raft_consensus.cc:2468] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:58.529958 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7da411d09de34ac88ce9e244b721fee5" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c" is_pre_election: true
I20260924 19:58:58.530488 20867 raft_consensus.cc:2468] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 0.
I20260924 19:58:58.531481 20658 leader_election.cc:304] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:58.532606 21400 raft_consensus.cc:2804] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:58:58.532908 21400 raft_consensus.cc:493] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:58:58.533183 21400 raft_consensus.cc:3060] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:58.544392 21400 raft_consensus.cc:515] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:58.547516 20574 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7da411d09de34ac88ce9e244b721fee5" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a4f50fc28c4616ba622ba51b0b2622"
I20260924 19:58:58.548044 20574 raft_consensus.cc:3060] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:58:58.555178 20574 raft_consensus.cc:2468] T 7da411d09de34ac88ce9e244b721fee5 P 30a4f50fc28c4616ba622ba51b0b2622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
I20260924 19:58:58.556485 20658 leader_election.cc:304] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a; no voters:
I20260924 19:58:58.557514 21432 raft_consensus.cc:2804] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:58:58.557978 21432 raft_consensus.cc:697] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [term 1 LEADER]: Becoming Leader. State: Replica: af20c71727a14e398a217d4623d1399a, State: Running, Role: LEADER
I20260924 19:58:58.558732 21432 consensus_queue.cc:237] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:58:58.561360 21400 leader_election.cc:290] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 1 election: Requested vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:58:58.563766 20867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7da411d09de34ac88ce9e244b721fee5" candidate_uuid: "af20c71727a14e398a217d4623d1399a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0055218dbcbd48b58018e9774e59e14c"
I20260924 19:58:58.564258 20867 raft_consensus.cc:3060] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:58:58.572487 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
@ 0x7f126e94899e std::__1::__function::__func<>::operator()() at ??:0
I20260924 19:58:58.592392 20867 raft_consensus.cc:2468] T 7da411d09de34ac88ce9e244b721fee5 P 0055218dbcbd48b58018e9774e59e14c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af20c71727a14e398a217d4623d1399a in term 1.
@ 0x5bd4e5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd31a std::__1::function<>::operator()() at ??:?
I20260924 19:58:58.614619 20206 catalog_manager.cc:5838] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a reported cstate change: term changed from 0 to 1, leader changed from <none> to af20c71727a14e398a217d4623d1399a (127.17.2.130). New cstate: current_term: 1 leader_uuid: "af20c71727a14e398a217d4623d1399a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } health_report { overall_health: UNKNOWN } } }
@ 0x7f12659aa81b kudu::Thread::SuperviseThread() at ??:0
@ 0x4569a0 __tsan_thread_start_func at ??:?
@ 0x7f126710b6db start_thread at ??:0
@ 0x7f126245661f clone at ??:0
W20260924 19:58:58.911095 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 4238ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:58:59.072496 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:58:59.183856 21450 raft_consensus.cc:493] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 19:58:59.185045 21450 raft_consensus.cc:515] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:58:59.193054 21450 leader_election.cc:290] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), 022c1898fcbe4856aa680c26047f6934 (127.17.2.188:35311)
W20260924 19:58:59.260825 20268 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 consensus state has no leader
W20260924 19:58:59.311143 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 4238ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:58:59.349277 20801 connection.cc:573] client connection to 127.17.2.190:44951 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:58:59.352345 20923 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.2.190:44951)
W20260924 19:58:59.353690 20923 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.17.2.190:44951)
W20260924 19:58:59.356665 20506 connection.cc:573] client connection to 127.17.2.190:44951 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:58:59.371237 20268 connection.cc:573] server connection from 127.0.0.1:38880 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:58:59.371836 20268 connection.cc:573] client connection to 127.17.2.190:44951 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:58:59.372160 20268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 4 similar messages]
W20260924 19:58:59.430287 20268 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.17.2.190:44951, user_credentials={real_user=slave}} blocked reactor thread for 57977.4us
W20260924 19:58:59.439589 21451 negotiation.cc:336] Failed RPC negotiation. Trace:
0924 19:58:59.354772 (+ 0us) reactor.cc:730] Submitting negotiation task for client connection to 127.17.2.190:44951 (local address 127.0.0.1:49150)
0924 19:58:59.414962 (+ 60190us) negotiation.cc:107] Waiting for socket to connect
0924 19:58:59.419707 (+ 4745us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":48970,"thread_start_us":46769,"threads_started":1}
W20260924 19:58:59.513619 20268 leader_election.cc:336] T 00000000000000000000000000000000 P f09f90f3506b48cd9c797dbbeffd0721 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 19:58:59.567029 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:58:59.628912 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 0.558s (SENT)
I20260924 19:58:59.629395 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:58:59.631929 21460 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260924 19:58:59.669762 21457 consensus_queue.cc:1048] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:58:59.704180 21457 consensus_queue.cc:1048] T 7da411d09de34ac88ce9e244b721fee5 P af20c71727a14e398a217d4623d1399a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:58:59.794147 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 5534ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:00.088490 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 5534ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:00.166075 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:59:00.223620 21468 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 19:59:00.224556 21468 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
W20260924 19:59:00.239054 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260924 19:59:00.243737 21468 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 19:59:00.244217 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 19:59:00.623049 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 6312ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:00.694049 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:00.879006 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 6312ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:00.999701 21457 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:01.000366 21457 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:59:01.002612 21457 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:59:01.004554 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 19:59:01.005131 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:59:01.005457 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:59:01.006286 21457 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:59:01.223443 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:01.339892 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 7102ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:01.593463 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 7102ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:01.819403 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:02.073891 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 7817ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:02.224622 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:02.358680 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 7817ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:02.475924 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 1.114s (SENT)
I20260924 19:59:02.476435 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:59:02.477983 21489 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 19:59:02.703282 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:02.828099 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 8582ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:03.089661 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 8582ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:03.262818 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:03.393046 20268 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 consensus state has no leader
I20260924 19:59:03.444782 21497 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 19:59:03.445389 21497 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:59:03.447810 21497 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 19:59:03.456818 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 19:59:03.558416 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 9313ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:03.714548 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:03.820912 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 9313ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:03.866201 21500 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:03.866727 21500 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:59:03.869289 21500 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:59:03.871342 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 19:59:03.873564 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:59:03.873899 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:59:03.874933 21500 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:04.257483 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 10044ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:04.320032 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:04.509052 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 10044ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:04.521449 21336 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:04.522188 21336 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:59:04.525241 21336 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 19:59:04.527782 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 19:59:04.528487 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 19:59:04.528868 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:59:04.529678 21336 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:04.827700 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:04.967360 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 10732ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:05.218734 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 10732ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:05.407567 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:05.680449 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 11442ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:05.837555 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:05.921761 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 11442ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:06.078219 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 1.311s (SENT)
I20260924 19:59:06.078907 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:59:06.080744 21518 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 19:59:06.357547 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:06.378805 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 12145ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:06.633044 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 12145ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:06.966367 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:07.111459 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 12856ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
I20260924 19:59:07.246634 21523 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:07.247102 21523 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:59:07.252046 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:59:07.252548 21523 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:59:07.264361 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:59:07.264780 20657 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:59:07.265432 21523 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:59:07.374068 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 12856ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:07.440177 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:59:07.653074 21540 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 19:59:07.653677 21540 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:59:07.656070 21540 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 19:59:07.659348 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260924 19:59:07.663574 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 19:59:07.824438 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 13597ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:07.900703 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:08.085208 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 13597ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:08.282796 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:08.543354 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 14308ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:08.759866 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:08.801254 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 14308ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:09.239492 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:09.260979 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 15024ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:09.519034 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 15024ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:09.740907 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:09.998399 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 15742ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:10.117358 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:10.262457 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 15742ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:10.525197 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:10.625545 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 1.475s (SENT)
I20260924 19:59:10.626046 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:59:10.627631 21570 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 19:59:10.724058 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 16486ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:10.903779 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:10.981969 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 16486ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:11.431145 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 17205ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:11.435833 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:11.613651 20268 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 consensus state has no leader
W20260924 19:59:11.674371 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 17205ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:11.909874 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:12.147068 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 17898ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:12.153759 21581 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:12.154452 21581 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:59:12.158946 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:59:12.160283 21581 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:59:12.161666 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:59:12.162213 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:59:12.162945 21581 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:59:12.380800 21586 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:12.381518 21586 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:59:12.387504 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 19:59:12.388846 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 19:59:12.389324 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:59:12.393555 21586 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 19:59:12.396198 21587 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:12.416055 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 17898ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 19:59:12.510394 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:59:12.548972 21580 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:12.549798 21580 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:59:12.554289 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 19:59:12.555510 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:59:12.555884 20657 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:59:12.556967 21580 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:59:12.557565 21580 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:59:12.774045 21594 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 19:59:12.775206 21594 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:59:12.779910 21594 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 19:59:12.793965 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260924 19:59:12.808459 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 19:59:12.880643 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 18639ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:13.095364 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:13.147199 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 18639ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:13.593066 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 19370ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:13.664145 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:13.853156 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 19370ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:14.113384 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:14.311261 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 20076ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 19:59:14.506259 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:14.557454 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 20076ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:14.995884 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:15.017576 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 20781ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:15.274907 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 20781ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:15.377444 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:15.731596 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 21498ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:15.733861 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 1.468s (SENT)
I20260924 19:59:15.734342 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:59:15.745319 21621 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 19:59:15.779515 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:15.991633 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 21498ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:16.192128 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:16.438918 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 22215ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:16.575634 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:16.686538 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 22215ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:16.989108 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:17.122314 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 22909ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:17.386598 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 22909ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:17.519497 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:17.850621 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 23610ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:18.085717 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:18.111356 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 23610ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:18.574159 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 24335ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:18.578411 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:18.845517 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 24335ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:19.129156 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:19.290488 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 25069ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:19.507496 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:59:19.508824 21649 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 19:59:19.509502 21649 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:59:19.512424 21649 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 19:59:19.522600 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260924 19:59:19.537926 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 25069ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:19.546108 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
I20260924 19:59:19.868674 21642 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:19.869350 21642 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:59:19.874387 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 19:59:19.874264 21642 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 19:59:19.875020 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 19:59:19.875345 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:59:19.875857 21642 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:19.997284 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 25761ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:20.058700 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:20.243197 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 25761ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:20.529356 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:20.705360 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 26466ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:20.990940 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 26466ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:21.102065 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:21.452436 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 27214ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:21.621241 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:21.712338 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 27214ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:22.193457 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 27936ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:22.237785 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:22.452118 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 27936ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:22.628396 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:22.896270 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 28675ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:23.055455 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:23.123557 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 28675ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:23.538901 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:23.582772 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 29347ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:23.818050 21686 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:23.818809 21686 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:59:23.823613 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 19:59:23.825757 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:59:23.826140 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:59:23.827066 21686 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:59:23.828109 21686 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:23.840467 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 29347ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:24.009334 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:24.292411 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 30063ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:24.487766 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:24.554114 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 30063ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:24.908758 21580 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:24.909315 21580 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:59:24.912094 21580 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:59:24.913587 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 19:59:24.914189 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:59:24.914491 20657 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:59:24.915213 21580 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:59:24.951926 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:25.007335 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 30777ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:25.276613 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 30777ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x472f7f __interceptor_pthread_setcancelstate at ??:?
@ 0x7fe80919384c boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 19:59:25.337280 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:25.715256 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 31500ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:25.932896 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:25.966658 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 31500ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:26.409653 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:26.421157 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 32190ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:26.679096 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 32190ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:26.828972 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:27.117333 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 32902ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:27.259897 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:27.382550 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 32902ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:27.840226 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 33606ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:27.847886 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:28.035370 20268 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 consensus state has no leader
W20260924 19:59:28.096521 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 33606ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:28.433468 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:28.454604 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 4.469s (SENT)
I20260924 19:59:28.455221 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:59:28.458492 21734 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 19:59:28.547013 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 34320ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:28.805501 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 34320ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:28.813409 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:29.215014 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:29.264901 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 35028ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:29.518385 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 35028ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:29.619951 21580 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:29.620430 21580 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:59:29.622099 21580 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:59:29.623893 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 19:59:29.624351 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:59:29.624723 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:59:29.625381 21580 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:59:29.795459 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:29.967319 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 35742ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:30.206888 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 35742ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:30.225740 21747 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 19:59:30.226263 21747 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
W20260924 19:59:30.230903 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260924 19:59:30.232699 21747 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 19:59:30.236101 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 19:59:30.351920 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:30.672034 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 36430ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:30.892453 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:30.934435 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 36430ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:31.389209 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 37158ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:31.482434 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:31.646385 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 37158ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:31.867647 21758 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:31.868280 21758 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:59:31.870947 21758 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:59:31.873039 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 19:59:31.873556 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:59:31.874010 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:59:31.874779 21758 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:31.941735 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:32.110263 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 37870ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:32.381430 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 37870ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:32.399930 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:32.837239 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 38605ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:32.881269 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:33.079365 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 38605ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:33.315074 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:33.535553 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 39303ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:33.717666 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:33.785370 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 39303ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:34.258237 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 40009ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:34.295477 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:59:34.502487 21780 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:34.503338 21780 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:59:34.505777 21780 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 19:59:34.508224 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 19:59:34.508925 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 19:59:34.509321 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:59:34.510092 21780 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:34.534251 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 40009ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:34.897787 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:35.002688 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 40757ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:35.284610 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 40757ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:35.349838 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:35.737823 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 41508ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:35.938773 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:36.001668 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 41507ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:36.457027 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 42225ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:36.459527 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:36.711190 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 42225ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:36.983276 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:37.172413 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 42934ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:37.432519 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 42934ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:37.573830 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:37.707535 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 3.736s (SENT)
I20260924 19:59:37.708050 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:59:37.709772 21812 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 19:59:37.881574 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 43656ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:37.957463 21799 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:37.958228 21799 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:59:37.962895 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 19:59:37.963611 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:59:37.963976 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:59:37.961162 21799 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:59:37.965101 21799 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:37.980257 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:59:38.121160 21580 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:38.121842 21580 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:59:38.125362 21580 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:59:38.127543 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 19:59:38.129168 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:59:38.129554 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:59:38.130231 21580 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:59:38.141294 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 43656ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:38.251791 21807 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:38.252269 21807 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:59:38.254758 21807 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 19:59:38.256556 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 19:59:38.257241 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 19:59:38.257601 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:59:38.258235 21807 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:38.459915 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:38.603929 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 44365ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:38.842295 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:38.866014 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 44364ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x472f7f __interceptor_pthread_setcancelstate at ??:?
@ 0x7fe80919384c boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 19:59:39.340159 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 45089ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:39.424117 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:59:39.499194 21826 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 19:59:39.499738 21826 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:59:39.505803 21826 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 19:59:39.507310 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260924 19:59:39.519614 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 19:59:39.596594 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 45089ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:40.014741 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:40.069048 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 45820ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:40.311775 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 45820ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:40.567143 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:40.764647 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 46535ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:41.017459 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 46535ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:41.138854 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:41.478408 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 47241ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:41.529937 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:41.731717 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 47241ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:42.148615 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:42.214231 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 47955ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:42.491412 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 47955ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:42.617374 21807 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:42.618247 21807 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:59:42.621580 21807 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 19:59:42.623425 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 19:59:42.624529 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 19:59:42.624867 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 19:59:42.625792 21807 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:42.670346 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:42.945086 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 48715ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:43.169199 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:43.200773 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 48715ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:43.654072 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 49424ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:43.764217 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:43.906502 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 49424ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:44.331423 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:44.375687 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 50130ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:44.472430 20771 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:44951 (90 consecutive failures): Network error: Failed to ping master at 127.17.2.190:44951: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 19:59:44.637372 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 50130ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:44.897782 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:44.954859 20622 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:44951 (91 consecutive failures): Network error: Failed to ping master at 127.17.2.190:44951: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 19:59:44.961612 20915 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:44951 (92 consecutive failures): Network error: Failed to ping master at 127.17.2.190:44951: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 19:59:45.079744 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 50860ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:45.259383 21799 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:45.259888 21799 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:59:45.268607 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:59:45.268782 21799 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:59:45.270269 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:59:45.270619 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:59:45.272606 21799 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:45.282943 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:45.364449 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 50860ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:45.463598 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 2.973s (SENT)
I20260924 19:59:45.464051 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 19:59:45.466104 21869 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 19:59:45.700137 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:45.802419 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 51588ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:46.046600 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 51588ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:46.100111 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:46.513388 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 52270ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:46.527714 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:46.767943 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 52270ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:47.021863 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:47.228338 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 52991ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:47.444288 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 52991ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:47.499411 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:59:47.506079 21580 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:47.506685 21580 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:59:47.512104 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 19:59:47.513792 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:59:47.514099 20657 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:59:47.514817 21580 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 19:59:47.515329 21580 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:59:47.888550 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 53667ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:47.900043 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:48.150826 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 53667ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:48.440928 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:48.611043 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 54374ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:48.817215 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:48.867391 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 54374ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:49.218245 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:49.330681 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 55091ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:49.613744 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:49.614409 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 55091ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:50.084533 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 55838ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:50.171051 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:50.361683 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 55838ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:50.727456 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:50.809592 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 56585ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:51.071107 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 56585ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:51.260365 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:51.538302 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 57294ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:51.675019 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:51.806519 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 57294ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:52.173628 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:52.270450 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 58031ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:52.554826 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 58031ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:52.690655 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:53.006068 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 58778ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:53.122360 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:53.270392 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 58778ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:53.500689 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:53.725641 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 59494ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:53.879132 21908 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:53.879853 21908 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 19:59:53.882094 21908 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:59:53.884987 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 19:59:53.886006 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 19:59:53.888619 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 19:59:53.890321 21908 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 19:59:53.973178 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 59494ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:54.035665 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:54.426576 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 60196ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:54.449117 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:54.698007 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 60196ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 19:59:54.820605 21943 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 19:59:54.821254 21943 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 19:59:54.824180 21943 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 19:59:54.827184 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260924 19:59:54.830777 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 19:59:55.001819 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:55.152644 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 60921ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:55.404943 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 60921ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:55.450928 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:55.875252 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 61628ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:56.059854 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:56.131832 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 61628ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:56.482667 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:56.581382 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 62355ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:56.838682 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 62355ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:56.935734 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:57.294106 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 63062ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:57.330724 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:57.550645 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 63062ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:57.807646 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:57.997807 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 63774ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:58.006970 20763 debug-util.cc:398] Leaking SignalData structure 0x7b08000e7780 after lost signal to thread 20632
W20260924 19:59:58.008309 20763 debug-util.cc:398] Leaking SignalData structure 0x7b08000cdaa0 after lost signal to thread 20766
W20260924 19:59:58.277834 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 63774ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:58.330168 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:58.773823 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:58.781633 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 64501ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:59.089195 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 64501ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:59.234925 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 19:59:59.408166 21953 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:59:59.408886 21953 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 19:59:59.413337 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:59:59.414166 21953 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 19:59:59.414278 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 19:59:59.414747 20657 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 19:59:59.415468 21953 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 19:59:59.590468 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 65312ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 19:59:59.672062 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 19:59:59.894026 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 65312ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:00.076771 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:00.402024 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 66117ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:00:00.420320 21968 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:00.421087 21968 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:00:00.427662 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:00:00.428815 21968 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:00:00.431298 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:00:00.431726 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:00:00.432951 21968 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:00:00.481990 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:00.686702 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 66117ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:00.810009 20268 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 consensus state has no leader
W20260924 20:00:01.021152 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:01.141211 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 66910ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:01.395407 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 66910ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:01.497865 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:01.868948 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 67619ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:02.048763 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:02.146679 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 67619ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:02.542260 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:02.610241 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 68370ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:02.881891 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 68370ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:02.986528 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:03.337584 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 69105ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:03.452237 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:03.586418 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 69105ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:03.856235 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:04.193816 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 69810ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:04.448765 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 69810ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:04.450621 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:04.920563 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 70672ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:04.996445 20315 debug-util.cc:398] Leaking SignalData structure 0x7b08000a7e20 after lost signal to thread 20250
W20260924 20:00:04.997315 20315 debug-util.cc:398] Leaking SignalData structure 0x7b080004e460 after lost signal to thread 20318
W20260924 20:00:05.032853 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:05.199707 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 70672ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:05.412949 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:05.697543 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 71423ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:05.803658 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:05.993412 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 71423ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:06.335628 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:06.448191 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 72216ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:06.549876 20616 debug-util.cc:398] Leaking SignalData structure 0x7b080003e960 after lost signal to thread 20480
W20260924 20:00:06.551090 20616 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3100 after lost signal to thread 20619
W20260924 20:00:06.753386 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 72216ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:06.938431 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:07.275097 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 72978ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:07.344354 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:07.576095 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 72978ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:07.861047 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:08.207146 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 73800ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:08.382347 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:08.526645 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 73800ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:08.772979 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:09.047739 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 74750ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:09.154994 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:09.339371 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 74750ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:09.728607 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:09.786473 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 75563ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:10.035986 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 75563ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:00:10.234571 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:10.490765 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 76259ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:10.802263 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 76259ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:10.824426 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:11.250300 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 77026ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:11.408392 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:11.508033 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 77026ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:11.968461 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 77731ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:12.014487 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:12.231843 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 77731ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:00:12.256386 22079 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:12.257071 22079 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:00:12.258982 22079 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:00:12.262751 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:00:12.263478 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:00:12.263849 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:00:12.264520 22079 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:00:12.484705 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:12.713195 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 78456ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:00:12.925644 22087 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:12.926367 22087 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:00:12.929559 22087 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:00:12.934075 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:00:12.935287 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:00:12.935860 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:00:12.937911 22087 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:00:12.960865 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 78455ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:12.994848 21148 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 18.169s (SENT)
W20260924 20:00:13.024070 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:13.423223 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 79184ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:13.530881 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:13.689738 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 79184ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:13.912526 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:14.142210 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 79913ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:14.309645 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:14.395653 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 79913ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:14.865937 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 80619ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:14.895970 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:15.117102 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 80619ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:15.297359 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:15.579238 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 81340ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:15.761845 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:15.840161 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 81340ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:16.291025 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 82063ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:16.295464 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:16.548321 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 82063ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:16.853467 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:00:16.899224 22038 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:16.899958 22038 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:00:16.902623 22038 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:00:16.904834 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:00:16.905416 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:00:16.905890 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:00:16.906674 22038 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:00:17.021028 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 82771ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:17.263592 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 82771ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:17.347388 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:17.738579 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 83487ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:17.739063 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:17.988169 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 83487ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:18.298303 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:18.444424 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 84211ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x4ae1e5 __tsan_atomic32_compare_exchange_strong at ??:?
@ 0x7fe8091b9092 base::subtle::NoBarrier_CompareAndSwap() at ??:0
@ 0x7fe8091b8ff2 kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x5bd4e5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd31a std::__1::function<>::operator()() at ??:?
W20260924 20:00:18.707098 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:18.730561 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 84211ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:19.183207 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:19.194082 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 84954ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:19.434571 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 84954ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:19.615479 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:19.881651 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 85658ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:20.015694 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:20.143313 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 85658ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:20.401161 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:20.594717 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 86366ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:20.866465 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 86366ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:20.999295 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:21.317907 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 87090ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:21.564499 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:21.576550 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 87090ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x4ae1e5 __tsan_atomic32_compare_exchange_strong at ??:?
@ 0x7fe8091b9092 base::subtle::NoBarrier_CompareAndSwap() at ??:0
@ 0x7fe8091b8ff2 kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x5bd4e5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd31a std::__1::function<>::operator()() at ??:?
W20260924 20:00:22.023912 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:22.029603 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 87800ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:22.321264 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 87800ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:22.538450 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:22.806634 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 88545ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:23.059757 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 88545ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:23.070868 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:23.506742 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:23.508517 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 89283ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:23.785742 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 89283ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:23.938709 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:24.242687 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 90009ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:24.411849 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:24.496131 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 90009ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:24.932677 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 90719ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:24.938251 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:25.189785 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 90719ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:00:25.213267 22173 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:25.214010 22173 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:00:25.216668 22173 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:00:25.218256 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:00:25.218820 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:00:25.219074 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:00:25.219735 22173 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:00:25.330860 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:25.715517 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:25.742926 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 91413ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:25.975497 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 91413ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:26.248935 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:26.428409 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 92199ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:26.637593 20315 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3fa0 after lost signal to thread 20250
W20260924 20:00:26.638372 20315 debug-util.cc:398] Leaking SignalData structure 0x7b0800098da0 after lost signal to thread 20318
W20260924 20:00:26.695950 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 92199ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:26.811939 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:27.168852 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 92919ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:00:27.293049 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:27.407763 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 92919ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x4ae1e5 __tsan_atomic32_compare_exchange_strong at ??:?
@ 0x7fe8091b9092 base::subtle::NoBarrier_CompareAndSwap() at ??:0
@ 0x7fe8091b8ff2 kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x5bd4e5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd31a std::__1::function<>::operator()() at ??:?
W20260924 20:00:27.702586 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:27.873703 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 93631ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:28.096462 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 93631ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:28.174700 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:28.552165 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:28.557448 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 94320ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:28.829931 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 94320ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:29.101016 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:29.308495 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 95053ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:29.567552 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 95053ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:29.637478 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:30.023247 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 95790ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:30.125221 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:30.288942 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 95790ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:30.671795 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:30.801014 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 96512ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:31.055240 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 96512ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:31.165093 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:31.169085 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 18.169s (SENT)
I20260924 20:00:31.169701 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 20:00:31.171615 22225 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 20:00:31.513022 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 97278ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:31.554706 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:31.779968 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 97278ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:00:31.857126 22224 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:31.857751 22224 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:00:31.860569 22224 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:00:31.862296 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:00:31.862881 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:00:31.863263 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:00:31.863974 22224 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:00:32.128342 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:32.242084 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 98003ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:32.495909 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 98003ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:32.627395 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:32.960501 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 98719ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:33.077533 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:33.210986 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 98719ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:33.616906 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:33.711463 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 99434ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:33.961155 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 99434ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:34.154315 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:34.421922 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 100184ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:34.676357 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 100184ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:34.738595 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:35.126205 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:35.132424 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 100899ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:35.394304 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 100899ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:35.546464 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:00:35.554482 22038 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:35.555116 22038 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:00:35.560259 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:00:35.562862 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:00:35.563222 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:00:35.564038 22038 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 20:00:35.564486 22038 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:00:35.606673 20273 maintenance_manager.cc:419] P f09f90f3506b48cd9c797dbbeffd0721: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033376
I20260924 20:00:35.741858 20272 maintenance_manager.cc:643] P f09f90f3506b48cd9c797dbbeffd0721: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.134s user 0.086s sys 0.036s Metrics: {"bytes_written":26960,"cfile_init":1,"drs_written":1,"lbm_read_time_us":361,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3504,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":67,"thread_start_us":7351,"threads_started":1,"wal-append.queue_time_us":7078}
I20260924 20:00:35.745191 20273 maintenance_manager.cc:419] P f09f90f3506b48cd9c797dbbeffd0721: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 4286 bytes on disk
I20260924 20:00:35.747479 20272 maintenance_manager.cc:643] P f09f90f3506b48cd9c797dbbeffd0721: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":354,"lbm_reads_lt_1ms":4}
W20260924 20:00:35.884035 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 101618ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:36.109026 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:36.126039 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 101617ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:36.584890 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 102349ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:36.711408 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:36.856343 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 102349ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:37.259716 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:37.390606 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 103080ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:37.660287 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 103080ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x472f7f __interceptor_pthread_setcancelstate at ??:?
@ 0x7fe80919384c boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:00:37.743142 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:38.101137 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 103883ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:38.242520 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:38.345563 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 103883ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:38.407927 20909 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1ca0 after lost signal to thread 20778
W20260924 20:00:38.409126 20909 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8d40 after lost signal to thread 20912
W20260924 20:00:38.664291 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:38.829303 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 104569ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:39.078019 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:39.110101 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 104569ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:00:39.432348 22278 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:39.432773 22278 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:00:39.434692 22278 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:00:39.438499 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:00:39.439010 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:00:39.439410 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:00:39.441691 22278 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:00:39.600721 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:39.608453 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 105333ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:39.934725 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 105333ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:40.171706 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:40.427740 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 106158ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:40.721208 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 106158ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:40.742501 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:41.204353 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 106944ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:41.280182 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:41.462795 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 106944ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:41.664610 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:41.906709 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 107686ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:42.166579 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 107686ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:42.232417 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:42.633579 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 108390ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:42.689080 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:42.892882 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 108390ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:43.145711 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:43.343652 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 109116ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:43.606304 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 109116ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:43.761196 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:44.038189 20909 debug-util.cc:398] Leaking SignalData structure 0x7b0800084ea0 after lost signal to thread 20778
W20260924 20:00:44.039435 20909 debug-util.cc:398] Leaking SignalData structure 0x7b08000fef40 after lost signal to thread 20912
I20260924 20:00:44.052847 22325 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:44.053553 22325 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:00:44.060221 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:00:44.061164 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:00:44.061123 22325 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 20:00:44.061563 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:00:44.062636 22325 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:00:44.073391 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 109830ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:00:44.107914 22323 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:44.108628 22323 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:00:44.116321 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:00:44.119266 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:00:44.119300 22323 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 20:00:44.119704 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:00:44.120918 22323 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:00:44.330013 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 109830ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:44.350396 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:44.495028 20771 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:44951 (208 consecutive failures): Network error: Failed to ping master at 127.17.2.190:44951: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:00:44.810521 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 110553ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:44.918503 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:44.986313 20914 heartbeater.cc:646] Failed to heartbeat to 127.17.2.189:40117 (75 consecutive failures): Timed out: Failed to ping master at 127.17.2.189:40117: Ping RPC to 127.17.2.189:40117 timed out after 1.000s (SENT)
W20260924 20:00:45.050835 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 110553ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:45.051685 20622 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:44951 (209 consecutive failures): Network error: Failed to ping master at 127.17.2.190:44951: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:00:45.495831 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 111274ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:45.505120 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:45.751870 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 111274ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:45.977883 21232 debug-util.cc:398] Leaking SignalData structure 0x7b0800077f20 after lost signal to thread 20959
W20260924 20:00:45.978312 21232 debug-util.cc:398] Leaking SignalData structure 0x7b0800078260 after lost signal to thread 20988
W20260924 20:00:45.978917 21232 debug-util.cc:398] Leaking SignalData structure 0x7b08000929e0 after lost signal to thread 21235
W20260924 20:00:46.094946 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:46.269582 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 111975ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:46.551427 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 111975ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:46.690070 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:46.998652 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 112775ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:47.246567 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:47.261230 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 112775ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:00:47.448314 22323 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:47.448815 22323 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:00:47.450799 22323 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:00:47.453696 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:00:47.454329 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:00:47.454671 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:00:47.455660 22323 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:00:47.739684 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 113484ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:47.753295 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:00:47.803033 22358 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 20:00:47.807564 22358 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
W20260924 20:00:47.843071 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
I20260924 20:00:47.839360 22358 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 20:00:47.868263 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:00:48.063654 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 113484ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:48.207294 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:48.536221 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 114287ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x4ae1e5 __tsan_atomic32_compare_exchange_strong at ??:?
@ 0x7fe8091b9092 base::subtle::NoBarrier_CompareAndSwap() at ??:0
@ 0x7fe8091b8ff2 kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x5bd4e5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd31a std::__1::function<>::operator()() at ??:?
W20260924 20:00:48.587842 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:48.837929 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 114287ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:49.094471 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:49.304210 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 115061ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:49.569085 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 115061ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:49.653328 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:49.996342 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 115792ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:50.057590 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:50.247696 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 115792ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:50.573908 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:50.687899 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 116471ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:50.946626 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 116471ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:51.084367 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:51.389806 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 117170ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:51.641780 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 117170ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:51.654495 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:52.090466 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 117865ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
I20260924 20:00:52.251732 21155 maintenance_manager.cc:419] P 022c1898fcbe4856aa680c26047f6934: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033343
W20260924 20:00:52.278604 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:52.360896 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 117865ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:00:52.385142 21154 maintenance_manager.cc:643] P 022c1898fcbe4856aa680c26047f6934: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.132s user 0.075s sys 0.032s Metrics: {"bytes_written":26960,"cfile_init":1,"drs_written":1,"lbm_read_time_us":324,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3355,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":67,"thread_start_us":725,"threads_started":1,"wal-append.queue_time_us":1106}
I20260924 20:00:52.388082 21155 maintenance_manager.cc:419] P 022c1898fcbe4856aa680c26047f6934: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 4286 bytes on disk
I20260924 20:00:52.389801 21154 maintenance_manager.cc:643] P 022c1898fcbe4856aa680c26047f6934: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":224,"lbm_reads_lt_1ms":4}
W20260924 20:00:52.824620 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 118584ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:52.852557 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:53.068907 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 118584ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:53.326495 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:53.529424 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 119292ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x4ae1e5 __tsan_atomic32_compare_exchange_strong at ??:?
@ 0x7fe8091b9092 base::subtle::NoBarrier_CompareAndSwap() at ??:0
@ 0x7fe8091b8ff2 kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x5bd4e5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd31a std::__1::function<>::operator()() at ??:?
W20260924 20:00:53.796831 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 119292ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:53.893661 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:54.230423 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 120020ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:54.283517 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:54.488952 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 120020ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:54.718772 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:54.937738 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 120712ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:55.126847 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:55.199461 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 120712ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:55.502633 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:55.660661 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 121423ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:55.906455 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 121423ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:56.092550 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:56.363699 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 122130ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:56.552121 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:56.609233 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 122130ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:57.010447 20315 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3ca0 after lost signal to thread 20250
W20260924 20:00:57.011327 20315 debug-util.cc:398] Leaking SignalData structure 0x7b0800095400 after lost signal to thread 20318
W20260924 20:00:57.067492 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 122832ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:57.082233 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:57.314852 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 122832ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:00:57.530249 22278 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:57.530807 22278 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:00:57.533394 22278 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:00:57.537222 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:00:57.538192 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:00:57.538529 20657 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:00:57.539669 22278 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:00:57.555425 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:57.785218 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 123538ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:58.005072 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:58.034808 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 123538ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:58.484122 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 124258ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:58.536404 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:58.729054 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 124258ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:58.942996 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:00:59.021967 22435 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:59.022642 22435 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:00:59.025678 22435 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:00:59.027448 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:00:59.028025 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:00:59.028431 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:00:59.029260 22435 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:00:59.177778 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 124952ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:00:59.307797 21148 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 11.467s (SENT)
W20260924 20:00:59.369007 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:59.420691 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 124952ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:00:59.738968 22450 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:00:59.739739 22450 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:00:59.742712 22450 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:00:59.744699 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:00:59.745380 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:00:59.745716 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:00:59.746368 22450 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:00:59.824563 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:00:59.875041 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 125644ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:00.127419 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 125644ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:00.423610 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:00.572785 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 126351ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:00.821031 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 126351ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:00.857503 20268 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 consensus state has no leader
W20260924 20:01:01.046464 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:01.270254 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 127044ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:01.518604 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 127044ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:01.528008 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:01.956432 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:01.968513 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 127742ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:01:02.235627 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 127742ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:02.354199 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:02.697034 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 128459ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:02.748307 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:02.948318 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 128459ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:03.174489 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:03.409773 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 129172ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:03.665792 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 129171ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:03.691001 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:04.113188 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 129889ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:04.184386 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:04.364142 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 129889ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:04.564097 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:04.818722 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 130587ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:05.069777 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 130587ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:05.139997 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:05.522264 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 131293ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:05.725648 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:05.782007 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 131293ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:06.180743 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:06.243078 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 132005ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:06.500293 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 132005ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:06.561240 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:06.959746 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:06.969862 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 132724ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:01:07.226634 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 132724ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:07.364696 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:07.689747 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 133450ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:07.920035 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:07.951738 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 133450ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:08.411197 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 134175ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:08.465250 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:08.655126 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 134175ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:09.067538 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:09.114485 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 134878ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:09.116931 20616 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0460 after lost signal to thread 20480
W20260924 20:01:09.118127 20616 debug-util.cc:398] Leaking SignalData structure 0x7b08000f68a0 after lost signal to thread 20619
W20260924 20:01:09.357296 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 134878ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:09.652256 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:09.822916 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 135580ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:10.080304 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 135580ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:10.262631 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:10.531037 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 136303ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:10.783581 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 136303ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:10.791873 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 11.467s (SENT)
I20260924 20:01:10.792551 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 20:01:10.794340 22556 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 20:01:10.881219 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:11.248203 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 137007ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:11.301887 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:11.489051 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 137007ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:01:11.559813 22540 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:11.560410 22540 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:01:11.562466 22540 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:01:11.564208 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:01:11.564852 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:01:11.565119 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:01:11.565727 22540 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:01:11.923833 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:11.945462 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 137712ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:12.204155 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 137712ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:12.517441 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:12.643517 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 138428ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:12.922298 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 138428ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:12.974432 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:13.379254 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 139146ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:13.580171 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:13.630190 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 139146ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:14.075793 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 139853ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:14.090790 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:01:14.094611 22589 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 20:01:14.095225 22589 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
W20260924 20:01:14.105142 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
I20260924 20:01:14.111269 22589 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 20:01:14.117530 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:01:14.341167 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 139853ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:14.522280 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:01:14.659783 22588 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:14.660380 22588 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:01:14.665140 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:01:14.665393 22588 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:01:14.667001 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:01:14.667385 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:01:14.668324 22588 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:01:14.800107 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 140564ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:01:14.859427 22546 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:14.860051 22546 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:01:14.862390 22546 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:01:14.864180 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:01:14.864815 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:01:14.865202 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:01:14.865882 22546 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:01:15.024166 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:15.050983 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 140564ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:15.500248 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 141274ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:01:15.506218 22540 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:15.506928 22540 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:01:15.510197 22540 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:01:15.512065 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:01:15.512718 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:01:15.513053 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:01:15.513731 22540 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:01:15.589097 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:15.760141 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 141274ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:16.190707 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:16.225457 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 141983ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:16.482134 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 141983ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:16.596542 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:16.921831 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 142705ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:17.083195 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:17.192248 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 142705ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:17.579690 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:17.639626 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 143416ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:17.892594 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 143415ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:18.178609 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:18.347226 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 144116ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:18.601456 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 144116ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:18.779505 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:19.040975 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 144824ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:19.294631 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:19.324038 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 144824ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:19.782625 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 145547ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:19.788224 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:20.026902 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 145547ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:20.329662 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:20.484428 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 146250ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:20.741245 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 146250ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:20.908834 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:21.197542 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 146964ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:21.346186 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:21.428125 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 146964ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:21.898806 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 147651ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:21.959091 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:01:22.058360 22639 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:22.059361 22639 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:01:22.063910 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:01:22.065062 22639 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:01:22.066978 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:01:22.067353 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:01:22.068077 22639 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:01:22.147061 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 147651ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:22.485713 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:22.604594 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 148371ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:01:22.718832 20616 debug-util.cc:398] Leaking SignalData structure 0x7b08000da840 after lost signal to thread 20480
W20260924 20:01:22.720014 20616 debug-util.cc:398] Leaking SignalData structure 0x7b08000e58c0 after lost signal to thread 20619
W20260924 20:01:22.837680 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 148371ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:22.954347 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:23.284344 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 149061ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:23.523222 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:23.529170 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 149061ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:01:23.758322 22651 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:23.759081 22651 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:01:23.761572 22651 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:01:23.763664 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:01:23.764267 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:01:23.764585 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:01:23.765304 22651 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:01:23.911787 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:23.984230 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 149752ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:24.240259 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 149752ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:24.462092 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:24.709532 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 150463ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:24.954766 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 150463ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:24.959441 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:01:25.067619 22689 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:25.068251 22689 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:01:25.072605 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:01:25.074432 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:01:25.074795 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:01:25.075564 22689 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 20:01:25.075645 22633 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:01:25.370798 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:25.409359 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 151178ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:25.666433 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 151178ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:25.836321 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:26.120416 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 151890ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:26.373853 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 151890ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:26.419296 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:26.830637 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 152597ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:26.935060 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:27.081859 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 152597ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:27.332858 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:27.537046 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 153305ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:27.792760 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 153305ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:27.802296 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:28.251564 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 154016ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:28.324200 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:28.517469 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 154016ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:28.891389 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:28.972867 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 154741ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:29.224347 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 154741ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:29.484719 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:29.658082 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 155448ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:29.912307 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 155448ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:30.093716 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:30.371382 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 156136ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:30.474499 21148 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 16.374s (SENT)
W20260924 20:01:30.616328 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 156136ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:30.626825 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:31.084012 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 156840ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:31.093258 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:31.343139 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 156840ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:31.597517 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:31.804533 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 157566ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:32.057055 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 157566ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:32.074812 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:32.471609 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:32.535359 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 158280ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:32.794425 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 158280ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:01:32.897110 22756 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:32.897836 22756 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:01:32.900579 22756 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:01:32.902282 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:01:32.902850 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:01:32.903250 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:01:32.904103 22756 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:01:33.023295 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:33.264714 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 159018ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:33.539655 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 159018ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:33.582158 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:33.992689 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 159763ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:34.138195 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:01:34.220903 22651 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:34.221570 22651 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:01:34.226575 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:01:34.227563 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:01:34.227916 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:01:34.228785 22651 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 20:01:34.229442 22651 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:01:34.244499 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 159763ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:34.605422 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:34.703491 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 160468ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:34.956452 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 160468ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:35.171031 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:35.444237 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 161180ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:35.697367 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 161180ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:35.789665 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:36.149914 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 161920ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:36.194396 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:36.413782 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 161920ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:36.583046 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:36.876966 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 162637ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:37.004819 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:37.140354 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 162637ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:37.423956 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:37.583577 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 163363ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:37.850243 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 163363ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:01:37.857240 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:38.298553 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 164073ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:38.323470 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:38.553942 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 164073ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:38.767361 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:01:38.778340 22712 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:38.778956 22712 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:01:38.783726 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:01:38.784296 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:01:38.784370 22712 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 20:01:38.784775 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:01:38.786017 22712 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:01:39.006492 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 164777ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:39.256242 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 164777ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:39.296339 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:39.712603 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 165480ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:39.764926 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:39.966027 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 165480ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:40.261126 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:40.413568 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 166189ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:40.673914 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 166189ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:40.756134 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:41.120863 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 166897ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:41.373968 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:41.378237 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 166897ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:41.817746 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 167601ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:41.848073 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:42.060667 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 167601ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:42.299045 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:42.525722 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 168284ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:42.706521 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:42.778134 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 168284ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:43.150411 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:43.232668 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 169001ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:43.476362 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 169001ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x4ae1e5 __tsan_atomic32_compare_exchange_strong at ??:?
@ 0x7fe8091b9092 base::subtle::NoBarrier_CompareAndSwap() at ??:0
@ 0x7fe8091b8ff2 kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x5bd4e5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd31a std::__1::function<>::operator()() at ??:?
W20260924 20:01:43.697155 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:43.932418 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 169700ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:44.181079 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 169700ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:44.310047 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:44.637353 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 170404ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:44.732810 20771 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:44951 (327 consecutive failures): Network error: Failed to ping master at 127.17.2.190:44951: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:01:44.810652 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:44.885874 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 170404ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:45.090381 20914 heartbeater.cc:646] Failed to heartbeat to 127.17.2.189:40117 (115 consecutive failures): Timed out: Failed to ping master at 127.17.2.189:40117: Ping RPC to 127.17.2.189:40117 timed out after 1.000s (SENT)
W20260924 20:01:45.228310 20621 heartbeater.cc:646] Failed to heartbeat to 127.17.2.189:40117 (115 consecutive failures): Timed out: Failed to ping master at 127.17.2.189:40117: Ping RPC to 127.17.2.189:40117 timed out after 1.000s (SENT)
W20260924 20:01:45.345314 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:45.351262 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 171109ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:45.603228 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 171109ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:01:45.623781 22854 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:45.624486 22854 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:01:45.626853 22854 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:01:45.629313 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:01:45.629900 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:01:45.630254 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:01:45.633149 22854 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:01:45.835430 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:46.048404 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 171826ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:46.304927 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 171826ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:46.360162 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:46.751186 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 172528ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:46.804491 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:46.852490 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 16.374s (SENT)
I20260924 20:01:46.853118 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 20:01:46.855124 22899 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 20:01:47.005359 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 172528ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:01:47.208124 22866 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:47.208737 22866 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:01:47.210995 22866 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:01:47.211423 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:47.213421 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:01:47.214497 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:01:47.214835 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:01:47.215596 22866 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:01:47.448050 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 173229ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:01:47.558059 22897 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:47.558692 22897 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:01:47.560950 22897 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:01:47.563025 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:01:47.563706 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:01:47.564085 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:01:47.564688 22897 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:01:47.605763 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:47.704937 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 173229ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:47.991927 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:48.162808 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 173928ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:48.392558 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 173928ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:48.418579 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:01:48.498538 22916 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 20:01:48.499060 22916 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 20:01:48.502120 22916 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 20:01:48.504683 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:01:48.508124 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:01:48.849480 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 174616ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:48.939931 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:49.109253 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 174616ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:49.552461 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:49.564602 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 175332ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:49.803738 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 175332ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:49.988031 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:50.264660 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 176027ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:50.453226 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:50.490585 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 176027ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:50.850559 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:50.889595 20763 debug-util.cc:398] Leaking SignalData structure 0x7b08000d3d40 after lost signal to thread 20632
W20260924 20:01:50.890573 20763 debug-util.cc:398] Leaking SignalData structure 0x7b08000d7d40 after lost signal to thread 20766
W20260924 20:01:50.955456 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 176714ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:51.215332 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 176714ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:51.329154 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:51.672113 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 177439ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:51.826558 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:51.918517 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 177439ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:52.269484 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:52.392896 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 178142ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:52.641774 20909 debug-util.cc:398] Leaking SignalData structure 0x7b08000f14c0 after lost signal to thread 20778
W20260924 20:01:52.642963 20909 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5b40 after lost signal to thread 20912
W20260924 20:01:52.645298 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 178142ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:52.754515 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:01:52.967556 22951 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:52.968374 22951 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:01:52.971971 22951 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:01:52.974243 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:01:52.977075 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:01:52.977417 20657 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:01:52.978097 22951 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:01:53.075073 20616 debug-util.cc:398] Leaking SignalData structure 0x7b08000ca720 after lost signal to thread 20480
W20260924 20:01:53.076367 20616 debug-util.cc:398] Leaking SignalData structure 0x7b08000f42e0 after lost signal to thread 20619
W20260924 20:01:53.117743 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 178869ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:53.235414 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:53.397392 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 178869ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:53.630522 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:53.838822 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 179620ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:54.105381 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 179620ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:54.144537 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:54.534279 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 180329ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:54.674350 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:54.793043 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 180329ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:55.081859 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:55.254720 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 181016ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:55.502049 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 181016ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:55.663381 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:55.964844 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 181725ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:56.221410 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 181725ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:56.256395 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:56.662931 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 182445ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:56.710734 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:56.918402 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 182445ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:57.165238 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:57.371882 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 183142ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:57.618449 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 183142ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:57.665966 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:01:57.818073 22995 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:57.818775 22995 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:01:57.821698 22995 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:01:57.823894 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:01:57.824477 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:01:57.824815 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:01:57.825500 22995 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:01:58.077307 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 183842ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:58.145422 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:01:58.303082 23009 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:01:58.303768 23009 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:01:58.308646 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:01:58.309239 23009 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:01:58.311393 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:01:58.311815 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:01:58.314942 23009 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:01:58.330106 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 183842ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x4ae1e5 __tsan_atomic32_compare_exchange_strong at ??:?
@ 0x7fe8091b9092 base::subtle::NoBarrier_CompareAndSwap() at ??:0
@ 0x7fe8091b8ff2 kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x5bd4e5 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x5bd31a std::__1::function<>::operator()() at ??:?
W20260924 20:01:58.678078 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:58.774959 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 184553ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:59.022584 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 184553ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:59.283970 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:01:59.487586 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 185246ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:59.723675 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 185245ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:01:59.847262 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:00.172338 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 185947ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:00.279003 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:00.427067 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 185947ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:00.788424 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:00.877753 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 186650ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:00.887565 20268 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 consensus state has no leader
W20260924 20:02:01.140659 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 186650ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:01.192734 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:01.570765 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 187364ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:01.765775 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:01.816705 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 187364ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:02.252835 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:02.288681 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 188040ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:02.538540 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 188040ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:02.838476 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:02.993374 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 188762ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:03.237737 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 188762ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:03.301332 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:03.685887 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 189461ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:03.828626 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:03.942236 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 189461ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:04.404620 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 190165ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:04.447916 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:04.643805 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 190165ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:04.835570 21148 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 16.332s (SENT)
W20260924 20:02:04.980849 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:05.105778 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 190867ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:05.349766 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 190867ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:05.588721 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:05.811945 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 191573ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:06.077167 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:06.098892 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 191573ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:06.462783 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:06.549964 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 192322ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:06.794476 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 192322ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:07.035447 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:07.268731 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 193018ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:07.424913 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:07.526290 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 193018ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:07.805299 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:07.981340 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 193750ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:08.196705 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:08.247685 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 193749ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x472f7f __interceptor_pthread_setcancelstate at ??:?
@ 0x7fe80919384c boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:02:08.597488 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:08.694916 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 194471ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:08.946074 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 194471ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:09.025020 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:09.387308 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 195169ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:09.635217 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 195169ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:09.639285 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:02:10.059211 23100 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:02:10.059945 23100 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:02:10.065634 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:02:10.069638 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:02:10.070044 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:02:10.071709 23123 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 20:02:10.072613 23100 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:02:10.074283 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 195858ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:10.103000 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:10.330061 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 195858ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:10.611649 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:10.788425 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 196553ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:11.044829 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 196553ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:11.161659 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:11.480502 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 197268ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:11.601686 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:11.723259 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 197268ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:12.067888 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:12.186944 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 197946ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:12.466176 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 197946ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x472f7f __interceptor_pthread_setcancelstate at ??:?
@ 0x7fe80919384c boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:02:12.542075 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:02:12.757735 23080 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:02:12.758409 23080 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:02:12.762164 23080 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:02:12.763767 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:02:12.764739 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:02:12.765151 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:02:12.765864 23080 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:02:12.934377 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 198690ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:13.109364 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:13.189029 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 198690ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:02:13.582531 23141 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:02:13.583375 23141 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:02:13.586520 23141 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:02:13.588940 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:02:13.589565 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:02:13.589907 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:02:13.590652 23141 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:02:13.607281 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:13.624218 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 199412ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:13.874310 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 199412ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:14.199498 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:14.337787 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 200098ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:14.599985 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 200098ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:14.790488 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:15.047412 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 200823ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:15.292313 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 200823ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:15.334401 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:15.748762 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 201515ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:15.897951 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:15.999788 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 201515ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:16.413416 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:16.447229 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 202223ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:16.683070 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 202223ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:16.853341 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:02:16.857735 23185 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:02:16.858239 23185 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:02:16.861202 23185 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:02:16.862646 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:02:16.863041 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:02:16.863426 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:02:16.863967 23185 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:02:17.150651 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 202906ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:17.324662 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:17.397882 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 202906ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:17.860044 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 203621ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:17.865340 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:18.125504 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 203621ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:18.336387 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:18.570561 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 204349ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:18.801985 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 204349ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:18.845041 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:19.281266 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 205025ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:19.395685 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:19.532801 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 205025ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:19.982918 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:19.984956 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 205756ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:20.251955 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 205756ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:20.401078 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:20.700176 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 206475ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:20.936254 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:20.954671 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 206475ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:21.171301 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 16.332s (SENT)
I20260924 20:02:21.171913 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 20:02:21.174060 23229 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 20:02:21.432471 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 207178ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:21.464669 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:21.682770 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 207178ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:21.981498 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:22.151753 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 207906ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:22.401420 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 207906ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:22.463811 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:22.634102 21232 debug-util.cc:398] Leaking SignalData structure 0x7b0800078240 after lost signal to thread 20959
W20260924 20:02:22.634477 21232 debug-util.cc:398] Leaking SignalData structure 0x7b0800073480 after lost signal to thread 20988
W20260924 20:02:22.635070 21232 debug-util.cc:398] Leaking SignalData structure 0x7b080009c1a0 after lost signal to thread 21235
W20260924 20:02:22.867995 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 208625ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:23.087047 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:23.133428 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 208625ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:23.548728 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:23.593099 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 209357ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:23.846767 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 209357ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:23.974040 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:24.303624 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 210070ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:24.402909 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:24.560717 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 210070ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:24.840824 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:25.033352 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 210784ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:25.276675 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:25.279682 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 210784ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:25.739799 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 211503ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:25.886085 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:25.995662 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 211503ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:26.408157 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:26.455353 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 212219ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:26.719195 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 212219ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:26.953125 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:27.179489 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 212942ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:27.420351 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 212942ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:27.464131 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:27.871287 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 213643ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:27.915190 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:28.134990 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 213643ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:28.302851 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:28.579769 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 214358ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:28.799096 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:28.838896 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 214358ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:29.283669 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 215062ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:29.422557 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:29.536315 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 215062ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:29.874447 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:29.977182 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 215759ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:30.241504 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 215759ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x472f7f __interceptor_pthread_setcancelstate at ??:?
@ 0x7fe80919384c boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:02:30.393369 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:02:30.615638 23312 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:02:30.616241 23312 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:02:30.618551 23312 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:02:30.620550 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:02:30.621083 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:02:30.621366 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:02:30.621937 23312 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:02:30.701956 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 216465ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:30.965713 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 216465ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:30.991953 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:31.420313 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 217189ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:31.481470 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:31.681067 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 217189ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:31.952031 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:32.138443 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 217904ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:32.390331 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 217904ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:32.396355 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:02:32.437393 23309 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:02:32.437983 23309 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:02:32.440652 23309 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:02:32.442030 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:02:32.443223 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:02:32.443554 20657 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:02:32.444272 23309 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:02:32.608155 23327 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:02:32.608840 23327 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:02:32.613250 23327 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:02:32.614760 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:02:32.615388 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:02:32.615770 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:02:32.616850 23327 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:02:32.847776 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 218614ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:33.000324 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:33.350976 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 218614ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:33.619300 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:02:33.693224 23346 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 20:02:33.693898 23346 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 20:02:33.704739 23346 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 20:02:33.717404 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:02:33.728319 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:02:33.814069 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 219574ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:34.068677 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 219574ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:34.107399 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:34.227788 20315 debug-util.cc:398] Leaking SignalData structure 0x7b080004f480 after lost signal to thread 20250
W20260924 20:02:34.228598 20315 debug-util.cc:398] Leaking SignalData structure 0x7b08000955c0 after lost signal to thread 20318
W20260924 20:02:34.488436 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:34.577447 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 220292ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:34.828887 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 220292ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:35.010829 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:35.272523 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 221052ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:35.337896 20616 debug-util.cc:398] Leaking SignalData structure 0x7b08000dff20 after lost signal to thread 20480
W20260924 20:02:35.339224 20616 debug-util.cc:398] Leaking SignalData structure 0x7b080003f700 after lost signal to thread 20619
W20260924 20:02:35.414141 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:35.541627 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 221052ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:35.946808 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:35.990904 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 221765ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:36.238332 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 221765ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:36.563977 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:36.706812 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 222461ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:36.965967 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 222461ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:37.059276 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:37.417659 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 223189ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:37.526381 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:37.670634 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 223189ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:37.959467 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:38.127017 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 223894ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:38.379927 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 223894ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:38.584158 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:38.834911 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 224603ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:39.095448 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 224603ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:39.114768 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:39.537438 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:39.553628 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 225319ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:39.833295 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 225318ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:40.003400 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:40.276217 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 226056ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:40.472853 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:40.528306 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 226056ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:40.860679 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:40.985409 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 226751ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:41.242522 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 226751ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:41.470321 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:41.726780 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 227466ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x472f7f __interceptor_pthread_setcancelstate at ??:?
@ 0x7fe809193810 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:02:41.882421 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:42.001739 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 227466ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:42.359386 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:02:42.420696 23408 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:02:42.421581 23408 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:02:42.427273 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:02:42.427842 23408 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:02:42.430227 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:02:42.430678 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:02:42.432492 23408 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:02:42.445526 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 228225ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:42.726969 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 228225ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:42.808229 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:43.184418 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 228951ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:43.227486 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:43.449445 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 228951ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:43.615702 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:43.915540 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 229673ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:44.173941 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 229673ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:44.175365 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:44.568524 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:44.648913 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 230397ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:02:44.908143 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 230397ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:44.966493 20771 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:44951 (446 consecutive failures): Network error: Failed to ping master at 127.17.2.190:44951: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:02:45.006844 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:45.197196 20914 heartbeater.cc:646] Failed to heartbeat to 127.17.2.189:40117 (155 consecutive failures): Timed out: Failed to ping master at 127.17.2.189:40117: Ping RPC to 127.17.2.189:40117 timed out after 1.000s (SENT)
W20260924 20:02:45.322248 20621 heartbeater.cc:646] Failed to heartbeat to 127.17.2.189:40117 (155 consecutive failures): Timed out: Failed to ping master at 127.17.2.189:40117: Ping RPC to 127.17.2.189:40117 timed out after 1.000s (SENT)
W20260924 20:02:45.361824 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 231131ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:45.465173 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:45.614040 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 231131ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:02:45.875828 23456 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:02:45.876466 23456 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:02:45.879380 23456 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:02:45.880744 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:02:45.881385 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:02:45.881716 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:02:45.882320 23456 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:02:46.017814 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:46.067696 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 231837ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:46.326939 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 231837ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:46.610950 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:46.778822 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 232550ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:47.013326 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:47.039211 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 232550ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:47.495398 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 233262ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:47.502405 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:47.757784 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 233262ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:47.927054 21148 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 14.221s (SENT)
W20260924 20:02:47.981098 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:48.217312 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 233981ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:48.362537 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:48.470638 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 233981ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:48.757404 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:48.942680 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 234694ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:49.202760 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 234694ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:49.333644 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:49.654897 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 235426ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:49.751397 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:49.914995 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 235426ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:50.217736 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:50.363946 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 236138ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:50.630369 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 236138ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:50.724720 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:51.098152 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 236853ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:02:51.195221 23501 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:02:51.196009 23501 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:02:51.199456 23501 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:02:51.201789 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:02:51.203123 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:02:51.203576 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:02:51.204375 23501 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:02:51.340801 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:51.343941 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 236853ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:51.820071 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 237567ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:51.841915 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:02:51.990749 23516 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:02:51.991351 23516 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:02:51.997044 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:02:51.999392 23516 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:02:52.002687 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:02:52.004447 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:02:52.010058 23518 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:02:52.081440 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 237567ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:52.446837 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:52.561705 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 238305ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:52.823001 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 238305ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:52.883787 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:53.298781 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 239046ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:53.484723 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:53.566612 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 239046ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:53.874470 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:54.001981 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 239790ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:54.251808 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 239790ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:54.436102 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:54.721128 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 240475ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:54.922813 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:54.972278 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 240475ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:55.305720 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:55.421790 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 241195ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:55.664413 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 241195ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:55.890017 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:56.127180 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 241888ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:56.367748 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 241888ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:56.487689 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:56.817067 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 242591ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:56.944566 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:57.078012 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 242591ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:57.339290 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:57.553660 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 243301ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:57.820564 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 243301ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:57.936247 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:58.282661 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 244044ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:58.553869 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:58.554682 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 244044ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:59.009953 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:59.015224 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 244778ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:59.269095 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 244778ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:59.470466 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:02:59.741294 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 245492ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:02:59.991814 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 245492ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:00.002919 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:03:00.322027 23481 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:00.322772 23481 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:00.325734 23481 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:03:00.327277 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:03:00.327878 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:03:00.328315 20658 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:03:00.329170 23481 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:03:00.455829 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 246215ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:00.491278 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:00.724326 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 246215ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:03:00.901774 20268 catalog_manager.cc:4845] Async tablet task AlterTable RPC for tablet 4477d9f1fa554d4b8ae762bfdcfedd44 (table default.table_77566e1a46ec4a4ba7dde05d53b7c5cc [id=c2d09462b2b14d1a862ef2b18152c4e1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4477d9f1fa554d4b8ae762bfdcfedd44 consensus state has no leader
W20260924 20:03:01.090780 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:01.192071 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 246948ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:01.446758 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 246948ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:01.549162 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:01.900750 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 247670ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:01.971551 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:02.152361 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 14.221s (SENT)
I20260924 20:03:02.153118 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 20:03:02.155241 23608 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 20:03:02.160772 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 247670ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:02.593084 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:02.640376 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 248384ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:03:02.672595 23600 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:02.673281 23600 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:02.675516 23600 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:03:02.677578 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:03:02.681113 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:03:02.681552 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:03:02.682385 23600 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 20:03:02.862244 23605 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:02.863023 23605 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:02.866807 23605 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:03:02.868072 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:03:02.868752 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:03:02.869094 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:03:02.869903 23605 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:03:02.896575 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 248384ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:03.061115 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:03.366701 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 249120ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:03.498643 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:03.635418 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 249120ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:03.930546 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:04.099426 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 249859ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:04.346069 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 249859ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:04.471001 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:04.813678 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 250569ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:04.880019 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:05.079988 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 250569ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:05.429375 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:05.566679 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 251303ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:05.827008 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 251303ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:05.827665 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:06.243191 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:06.299036 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 252050ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:06.559072 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 252050ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:06.812546 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:07.013258 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 252782ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:07.227567 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:07.262531 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 252782ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:07.731971 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 253486ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:07.841988 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:07.999073 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 253486ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:08.357673 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:08.468416 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 254222ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:08.715761 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 254222ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:08.966097 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:09.179975 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 254939ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:09.343494 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:09.425029 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 254939ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:09.889627 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 255648ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:09.925202 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:10.148841 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 255648ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:10.499580 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:10.606062 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 256372ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:10.854614 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 256372ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:11.020736 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:11.303678 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 257078ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:11.478825 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:11.595389 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 257078ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:11.906682 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:12.053794 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 257819ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:12.314828 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 257819ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:12.414104 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:12.786903 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 258538ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:03:13.003815 23677 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260924 20:03:13.004760 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:03:13.004554 23677 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:13.008551 23677 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:03:13.009753 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:03:13.010432 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:03:13.010769 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:03:13.011559 23677 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:03:13.044723 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 258538ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:13.455581 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:13.516410 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 259268ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:13.777179 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 259268ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:14.051383 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:03:14.202600 23721 raft_consensus.cc:493] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 1feab783ba9d4112932c72545e55ea19)
I20260924 20:03:14.203352 23721 raft_consensus.cc:515] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1feab783ba9d4112932c72545e55ea19" member_type: VOTER last_known_addr { host: "127.17.2.190" port: 44951 } } peers { permanent_uuid: "f09f90f3506b48cd9c797dbbeffd0721" member_type: VOTER last_known_addr { host: "127.17.2.189" port: 40117 } } peers { permanent_uuid: "022c1898fcbe4856aa680c26047f6934" member_type: VOTER last_known_addr { host: "127.17.2.188" port: 35311 } }
I20260924 20:03:14.206331 23721 leader_election.cc:290] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951), f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117)
W20260924 20:03:14.210307 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:03:14.215653 21146 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 1feab783ba9d4112932c72545e55ea19 (127.17.2.190:44951): Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:03:14.232827 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 260000ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:14.465557 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:14.494056 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 260000ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:14.855763 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:14.959134 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 260717ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:15.218494 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 260717ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:15.439963 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:03:15.466878 23716 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:15.467626 23716 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:15.470168 23716 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:03:15.472433 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:03:15.473088 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:03:15.473440 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:03:15.474138 23716 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:03:15.672535 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 261442ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:15.929841 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 261442ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:15.982368 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:16.381623 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 262153ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:16.502620 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:16.642313 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 262153ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:03:16.716073 23673 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:16.716660 23673 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:16.719100 23673 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:03:16.720996 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:03:16.721562 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:03:16.721935 20657 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:03:16.722968 23673 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:03:16.923774 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:17.090464 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 262865ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:17.342747 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:17.351037 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 262865ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:17.799458 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 263574ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:17.965008 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:18.058889 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 263574ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:18.470934 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:18.518543 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 264282ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:18.770520 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 264282ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:18.877158 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:19.220707 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 264994ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:19.464949 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 264994ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:19.466777 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:19.920562 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 265688ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:19.991758 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:20.165804 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 265688ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:20.578902 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:20.635401 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 266389ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:20.882392 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 266389ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:20.972319 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:21.352412 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 267106ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:21.508514 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:21.554553 20909 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8220 after lost signal to thread 20778
W20260924 20:03:21.555923 20909 debug-util.cc:398] Leaking SignalData structure 0x7b08000da020 after lost signal to thread 20912
W20260924 20:03:21.626510 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 267106ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:22.042716 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:22.095232 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 267850ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:22.334597 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 267850ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x472f7f __interceptor_pthread_setcancelstate at ??:?
@ 0x7fe809193810 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
W20260924 20:03:22.447726 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:22.808813 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 268558ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:22.886212 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:23.044955 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 268558ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:23.315805 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:23.510690 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 269269ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:03:23.627606 23793 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:23.628787 23793 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:23.631502 23793 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
W20260924 20:03:23.636160 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:03:23.636868 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:03:23.637193 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:03:23.638213 23793 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:03:23.763000 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 269269ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:23.793550 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:24.213893 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 269986ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:24.312819 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:24.466300 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 269986ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:24.788592 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:24.927026 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 270690ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:25.180284 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 270689ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:25.272262 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:25.647197 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 271404ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:25.669755 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:25.901957 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 271404ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:26.069438 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:26.362953 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 272125ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:26.554576 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:26.622566 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 272125ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:27.078819 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:27.081861 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 272846ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:27.342464 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 272846ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:27.491302 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:27.780647 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 273566ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:28.016856 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:28.036269 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 273566ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:28.483671 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 274259ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:28.491629 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:28.754773 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 274259ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:28.927353 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:29.203632 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 274978ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:29.424973 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:29.442452 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 274978ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:03:29.624389 23850 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:29.625037 23850 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:29.627290 23850 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:03:29.629153 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:03:29.630280 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:03:29.630838 20657 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:03:29.631727 23850 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:03:29.892823 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:29.893213 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 275666ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:30.122687 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 275666ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:30.407626 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:30.589843 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 276346ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:30.849566 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 276346ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:30.982097 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:31.200313 21148 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 16.993s (SENT)
W20260924 20:03:31.318241 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 277073ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:31.486912 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:31.609062 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 277073ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:03:31.708832 23872 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:31.709477 23872 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:03:31.714087 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:03:31.715330 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:03:31.715698 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:03:31.713922 23872 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
I20260924 20:03:31.716912 23872 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:03:31.981434 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:32.072064 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 277832ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:32.329221 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 277832ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:32.550678 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:32.794363 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 278552ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:32.964996 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:33.030264 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 278552ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:33.412207 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:33.483414 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 279254ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:33.744861 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 279254ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:33.865746 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:34.208452 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 279968ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:34.466938 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:34.473946 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 279968ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:34.927925 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 280697ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:35.043001 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:03:35.147951 23899 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:35.148375 23899 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:35.150480 23899 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:03:35.152563 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:03:35.152971 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:03:35.153375 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:03:35.154127 23899 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:03:35.184252 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 280697ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:35.578576 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:35.651404 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 281407ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:35.903445 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 281407ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:36.007301 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:36.349156 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 282127ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:36.470685 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:36.608091 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 282127ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:36.915557 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:37.063223 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 282831ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:37.315035 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 282831ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:37.537320 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:37.771809 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 283538ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:37.928382 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:38.012030 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 283538ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:38.405412 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:38.488169 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 284235ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:38.757700 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 284235ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:38.809551 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:39.195470 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 284981ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:39.362946 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:39.424001 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 284981ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:39.874030 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 285647ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:39.890770 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260924 20:03:39.946713 23941 raft_consensus.cc:493] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:39.947530 23941 raft_consensus.cc:515] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
W20260924 20:03:39.953109 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
W20260924 20:03:39.953804 20804 leader_election.cc:343] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0d49e3b1113b4614b359e9922e7fe55f
I20260924 20:03:39.954146 20804 leader_election.cc:304] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0055218dbcbd48b58018e9774e59e14c; no voters: 30a4f50fc28c4616ba622ba51b0b2622, af20c71727a14e398a217d4623d1399a
I20260924 20:03:39.955736 23946 raft_consensus.cc:2749] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 20:03:39.962100 23941 leader_election.cc:290] T 0d49e3b1113b4614b359e9922e7fe55f P 0055218dbcbd48b58018e9774e59e14c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), af20c71727a14e398a217d4623d1399a (127.17.2.130:38719)
I20260924 20:03:40.040853 23933 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:40.041517 23933 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:40.043946 23933 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:03:40.046428 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:03:40.047638 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:03:40.048003 20509 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:03:40.048766 23933 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:03:40.123148 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 285647ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:40.370128 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:40.578577 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 286346ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:40.843307 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 286346ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:40.926993 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:41.292012 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 287066ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:41.308597 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:41.553606 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 287066ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:41.880249 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:42.008204 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 287777ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:42.244299 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 287777ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:42.325448 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:42.589576 20763 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8780 after lost signal to thread 20632
W20260924 20:03:42.591187 20763 debug-util.cc:398] Leaking SignalData structure 0x7b08000adce0 after lost signal to thread 20766
W20260924 20:03:42.730927 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 288468ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:42.802351 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:42.975111 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 288468ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:43.310451 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:43.426967 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 289199ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:43.676937 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 289199ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:43.917761 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:44.119207 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 289900ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:44.366178 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 289900ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:44.532464 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:44.837210 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 290589ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:44.974677 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:45.096850 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 290589ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:45.208880 20915 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:44951 (565 consecutive failures): Network error: Failed to ping master at 127.17.2.190:44951: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:03:45.250963 20771 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:44951 (565 consecutive failures): Network error: Failed to ping master at 127.17.2.190:44951: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:03:45.392489 20622 heartbeater.cc:646] Failed to heartbeat to 127.17.2.190:44951 (565 consecutive failures): Network error: Failed to ping master at 127.17.2.190:44951: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection refused (error 111)
W20260924 20:03:45.510044 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:45.576021 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 291320ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:45.838095 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 291320ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:46.052452 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:46.292171 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 292061ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:46.544936 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 292061ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:46.653477 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:46.992833 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 292768ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:47.176188 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:47.242027 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 292768ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:47.709383 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 293465ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:47.774911 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:47.972293 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 293465ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:03:48.080215 24000 raft_consensus.cc:493] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:48.080808 24000 raft_consensus.cc:515] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:48.084517 24000 leader_election.cc:290] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:03:48.085842 20658 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 30a4f50fc28c4616ba622ba51b0b2622 (127.17.2.129:33667): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
W20260924 20:03:48.086545 20657 leader_election.cc:343] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: e6dcbdac79d04942b3ac59dc11a0ed3c
I20260924 20:03:48.086931 20657 leader_election.cc:304] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af20c71727a14e398a217d4623d1399a; no voters: 0055218dbcbd48b58018e9774e59e14c, 30a4f50fc28c4616ba622ba51b0b2622
I20260924 20:03:48.087713 24000 raft_consensus.cc:2749] T e6dcbdac79d04942b3ac59dc11a0ed3c P af20c71727a14e398a217d4623d1399a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260924 20:03:48.197487 21148 leader_election.cc:336] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f09f90f3506b48cd9c797dbbeffd0721 (127.17.2.189:40117): Timed out: RequestConsensusVote RPC to 127.17.2.189:40117 timed out after 16.993s (SENT)
I20260924 20:03:48.198120 21148 leader_election.cc:304] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 022c1898fcbe4856aa680c26047f6934; no voters: 1feab783ba9d4112932c72545e55ea19, f09f90f3506b48cd9c797dbbeffd0721
I20260924 20:03:48.200400 24020 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 022c1898fcbe4856aa680c26047f6934 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260924 20:03:48.211992 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:48.431910 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 294195ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:48.676518 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 294195ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:48.735597 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:49.123914 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 294900ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:49.263273 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:49.368291 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 294900ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:49.821368 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 295591ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:49.878140 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:50.074646 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 295591ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:50.467824 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:50.541133 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 296298ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:50.820174 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 296298ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:50.986995 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:51.276139 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 297043ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:51.420809 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:51.548264 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 297043ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:51.969868 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:51.987820 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 297771ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:52.223773 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 297771ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:52.477728 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:52.694376 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 298447ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:52.955283 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 298447ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:52.960434 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:53.376160 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:53.393239 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 299179ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:53.650230 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 299179ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:53.995733 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:54.093791 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 299873ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
W20260924 20:03:54.334223 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 299873ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
I20260924 20:03:54.470656 24078 raft_consensus.cc:493] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:03:54.471388 24078 raft_consensus.cc:515] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a4f50fc28c4616ba622ba51b0b2622" member_type: VOTER last_known_addr { host: "127.17.2.129" port: 33667 } } peers { permanent_uuid: "af20c71727a14e398a217d4623d1399a" member_type: VOTER last_known_addr { host: "127.17.2.130" port: 38719 } } peers { permanent_uuid: "0055218dbcbd48b58018e9774e59e14c" member_type: VOTER last_known_addr { host: "127.17.2.131" port: 41601 } }
I20260924 20:03:54.474186 24078 leader_election.cc:290] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af20c71727a14e398a217d4623d1399a (127.17.2.130:38719), 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601)
W20260924 20:03:54.475888 20509 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer af20c71727a14e398a217d4623d1399a (127.17.2.130:38719): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
W20260924 20:03:54.478572 20508 leader_election.cc:343] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0055218dbcbd48b58018e9774e59e14c (127.17.2.131:41601): Not found: Tablet not found: 0c40348984314595b6e6e49c4f03f35e
I20260924 20:03:54.478883 20508 leader_election.cc:304] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 30a4f50fc28c4616ba622ba51b0b2622; no voters: 0055218dbcbd48b58018e9774e59e14c, af20c71727a14e398a217d4623d1399a
I20260924 20:03:54.479523 24078 raft_consensus.cc:2749] T 0c40348984314595b6e6e49c4f03f35e P 30a4f50fc28c4616ba622ba51b0b2622 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 20:03:54.485628 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260924 20:03:54.788218 20925 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311, num_attempts: 759) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection reset by peer (error 104)
W20260924 20:03:54.788887 20925 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311, num_attempts: 759) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection reset by peer (error 104)
W20260924 20:03:54.795660 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 300557ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x456ae0 __tsan::CallUserSignalHandler() at ??:?
@ 0x4590c8 rtl_sigaction() at ??:?
@ 0x7fe801232980 (unknown) at ??:0
@ 0x7fe801231d50 __GI___nanosleep at ??:0
@ 0x47b8c4 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
F20260924 20:03:54.806605 20931 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_b9502ead412146d497587b0c995526fa on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311, num_attempts: 759) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection reset by peer (error 104)
*** Check failure stack trace: ***
*** Aborted at 1790280234 (unix time) try "date -d @1790280234" if you are using GNU date ***
F20260924 20:03:54.813638 20934 master-stress-test.cc:299] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_d7b4827d2d284d37b72f600e797d818d on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.17.2.190:44951,127.17.2.189:40117,127.17.2.188:35311, num_attempts: 759) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.17.2.190:44951: connect: Connection reset by peer (error 104)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000440a) received by PID 17418 (TID 0x7f8bfe2fe700) from PID 17418; stack trace: ***
@ 0x2b0620 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
W20260924 20:03:54.835523 20923 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.17.2.190:44951)
@ 0x2b2b03 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f8c0fc0c980 (unknown) at ??:0
@ 0x7f8c0adb6e87 gsignal at ??:0
@ 0x7f8c0adb87f1 abort at ??:0
@ 0x2b6416 __interceptor_abort at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f8c0de54d74 google::LogMessage::SendToLog() at ??:0
@ 0x7f8c0de55920 google::LogMessage::Flush() at ??:0
@ 0x7f8c0de5aa5b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x35e444 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38bc3e _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846
@ 0x38baf1 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38b23b std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2b04e0 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
W20260924 20:03:54.917550 20268 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
@ 0x7f8c0fc016db start_thread at ??:0
@ 0x7f8c0ae9961f clone at ??:0
W20260924 20:03:55.094321 20254 kernel_stack_watchdog.cc:198] Thread 20270 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:249 for 300557ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x459ad4 __tsan::ProcessPendingSignals() at ??:?
@ 0x459d85 __tsan::ScopedInterceptor::~ScopedInterceptor() at ??:?
@ 0x47b851 __interceptor_nanosleep at ??:?
@ 0x7fe809193835 boost::core::sp_thread_sleep() at ??:0
@ 0x7fe8091937df boost::detail::yield() at ??:0
@ 0x7fe8091b8fcc kudu::rw_semaphore::lock() at ??:0
@ 0x7fe8091b8f5a kudu::rw_spinlock::lock() at ??:0
@ 0x7fe8089ddf49 std::__1::lock_guard<>::lock_guard() at ??:0
@ 0x7fe8089afa7e kudu::master::TableInfo::RemoveTask() at ??:0
@ 0x7fe8089ae8f4 kudu::master::RetryingTSRpcTask::UnregisterAsyncTask() at ??:0
@ 0x7fe8089af36a kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fe808a0aad2 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7fe808a0aa62 _ZNSt3__18__invokeIRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
@ 0x7fe808a0a9f2 _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_EEEvDpOT_ at ??:0
@ 0x7fe808a0a9b2 _ZNSt3__110__function12__alloc_funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0
@ 0x7fe808a0964e _ZNSt3__110__function6__funcIZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_NS_9allocatorIS5_EEFvvEEclEv at ??:0